docs: align operations runbooks with manual releases

This commit is contained in:
2026-08-20 18:35:26 +03:30
parent 892d79b6c3
commit 789f60e2ca
2 changed files with 29 additions and 29 deletions

View File

@@ -16,10 +16,10 @@ Before connecting to the host, record in the change ticket/release record:
- operator names, maintenance window, rollback/abandonment condition, and emergency contact
- DNS zone owner and current `A`/`AAAA`/CAA records for `zarinpal.amiirkhl.ir`
- SSH source allowlist and at least two tested operator public keys
- selected rootless or dedicated rootful Docker model (a remaining decision in the platform spec)
- private registry hostname and read-only production pull identity
- dedicated deployment username that will receive Docker-group access
- approved deployment, backend, and frontend repository URLs and expected nested paths
- object-storage bucket/endpoint, retention capability, backup credentials, and separately escrowed encryption key
- approved release manifest containing exact image digests and all three repository SHAs
- target `main` branches and an empty location for the first three-hash release record
- secret inventory and owner; never put values in the ticket, shell history, or this repository
Do not change DNS before the host passes local and direct-IP validation. Do not reuse Qlockify credentials, certificates, volumes, backup paths, or deploy keys.
@@ -58,23 +58,23 @@ Evidence: OS release/kernel, update result, time-sync state, two successful key
## Phase 3: install and constrain Docker
1. Install Docker Engine and the Compose plugin only from Docker's approved repository for the selected OS; pin/record installed versions.
2. Apply the approved rootless or dedicated rootful design. If the deploy user joins the `docker` group, record that it is effectively root-equivalent and restrict that account accordingly.
2. Add only the dedicated trusted deploy user to the `docker` group. Record that this access is effectively root-equivalent and restrict the account accordingly.
3. Configure the daemon's bounded logging policy before starting application containers.
4. Ensure Docker starts after reboot and that no unauthenticated TCP daemon socket is enabled.
5. Log in to the private registry using the production read-only pull identity. Store credentials in the approved host credential mechanism, not the deployment checkout or shell history.
6. Pull a harmless pinned test image, verify its digest, run it without published ports, remove the test container, and retain the image only if policy allows.
5. Pull a harmless pinned public test image, verify its tag/digest, run it without published ports, then remove the test container.
6. Confirm the deploy user can run Docker Compose without using a broader shared administrator account.
Evidence: Engine/Compose versions, daemon security configuration, socket ownership, log driver/options, registry identity scope, and digest verification.
Evidence: Engine/Compose versions, daemon security configuration, socket/group ownership, deploy-user membership, log driver/options, and test-image result.
## Phase 4: install the deployment control plane
1. Create a dedicated deploy machine identity with read access to the deployment repository and no write/push permission. The production host does not need backend/frontend source checkouts when deploying immutable images.
2. Clone the deployment repository into `/srv/zarinpal-analytics/current` or the approved release-control path. Pin/checkout the reviewed deployment SHA from the release manifest without destructive synchronization of unrelated paths.
3. Validate repository ownership, remote URL, current SHA, and clean status.
1. Create one server Git identity/key with read-only access to all three repositories and no write/push permission.
2. Clone the deployment repository into the approved deployment root, then clone backend and frontend into its exact ignored nested paths.
3. Validate ownership, remote URL, `main` branch, current SHA, clean status, and independent `.git` directory for every repository.
4. Materialize protected non-secret configuration and secret files outside the checkout. Set owner/mode before writing secret values; verify consumers receive only required files.
5. Install the production release manifest and verify its signature/checksum, exact target environment, previous release pointer, image digests, and configuration checksum.
6. Pull every manifest image by digest. Fail if a digest is missing, mutable, untrusted, or blocked by the vulnerability/provenance gate.
7. Render/validate Compose configuration in a way that does not print secret values into logs. Confirm only Nginx publishes 80/443 and operator endpoints bind only to loopback/private paths.
5. Fetch each repository and confirm it can fast-forward cleanly to `origin/main`; do not reset, build, or start yet.
6. Record the three initial commit hashes and selected infrastructure image tags in a non-secret draft release record.
7. Render/validate Compose configuration without printing secret values. Confirm only Nginx publishes 80/443.
Do not run `docker compose up`, migrations, import, credential generation, restore, or DNS cutover during this phase.
@@ -94,14 +94,14 @@ If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/c
1. Acquire the production deployment lock and record the operator/release id.
2. Start PostgreSQL and Redis only. Verify named-volume targets, authentication, resource limits, and health.
3. Run the one-off migration at the candidate backend digest. Confirm exit zero and the exact expected migration version.
3. Run the one-off migration from the recorded backend checkout/build. Confirm exit zero and the exact expected migration version.
4. Start backend and worker. Verify liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
5. Start frontend and verify the static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
6. Start/reload Nginx and run public generic health, canonical redirect, security-header, login-page, and authenticated smoke checks.
7. Verify Prometheus targets/dashboards and test notification delivery before relying on the 15-minute release watch.
8. Record the active release manifest only after all postflight checks pass.
7. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
8. Record the active three-hash release only after all postflight checks pass.
Failure before the active-manifest switch restores the prior/empty known state. Do not improvise a database down migration.
Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration.
## Phase 7: one-time data and demo setup
@@ -121,22 +121,22 @@ Never add the source archive or credentials to Git, a container image, backup pa
1. Run the first encrypted logical PostgreSQL backup.
2. Verify archive listing/checksum, encrypted local artifact, remote upload, remote checksum, manifest, and retention counts.
3. Restore that backup to a clean non-production destination and complete application/tenant/analytics validation within the two-hour RTO.
4. Test certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age alerts.
4. Perform and record the MVP manual certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age checks.
5. Record current release, DNS, certificate, backup id, achieved RPO/RTO, remaining exceptions, and on-call ownership.
6. Remove temporary bootstrap credentials/files and close provider rules that are no longer needed.
The host is not considered production-ready until restore rehearsal and alert delivery pass.
The host is not considered MVP-ready until restore rehearsal and every manual operator check passes. Prometheus/Grafana and automated alert delivery are the required next phase before AI work.
## Stop/rollback conditions
Stop without progressing when:
- recovery console or second-operator SSH access is not proven
- resolved paths, host identity, environment, or release manifest do not match
- an image is not digest-pinned or provenance/vulnerability checks fail
- resolved paths, host identity, repository remotes/branches, environment, or release record do not match
- a repository is dirty/diverged unexpectedly or a build/source dependency cannot be verified
- unexpected ports are externally reachable
- DNS points somewhere unexpected or certificate names/chain are invalid
- migration, readiness, authenticated smoke, tenant isolation, worker/SSE, monitoring, backup, or restore rehearsal fails
- migration, readiness, authenticated smoke, tenant isolation, worker/SSE, manual operational checks, backup, or restore rehearsal fails
- secrets appear in output, files have broad permissions, or the deployment checkout is dirty unexpectedly
Rollback DNS to the recorded prior values when the new host cannot pass the cutover window. Preserve logs and manifests, revoke any exposed credential, and do not delete the failed host until evidence and required backups are secured.
@@ -147,11 +147,11 @@ Rollback DNS to the recorded prior values when the new host cannot pass the cuto
- [ ] Console plus two key-only operator paths proven
- [ ] Firewall external tests show only approved ports
- [ ] Docker/Compose versions and privilege model recorded
- [ ] Deployment SHA and release manifest verified
- [ ] Three repository remotes/branches/SHAs and release record verified
- [ ] Secrets outside Git with restrictive ownership/mode
- [ ] TLS issuance and renewal tested
- [ ] Migration, API, worker, frontend/font, Nginx, SSE, and tenant smoke pass
- [ ] One-time import reconciles and demo credentials stay outside logs/Git
- [ ] Monitoring targets and alert delivery pass
- [ ] MVP manual health/log/disk/certificate/backup/queue checks pass
- [ ] Encrypted off-host backup and clean restore rehearsal pass within RTO
- [ ] Active/previous release ids and operator handoff recorded