From 789f60e2cab627620ed40566db3d56c62145ab41 Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 18:35:26 +0330 Subject: [PATCH] docs: align operations runbooks with manual releases --- docs/runbooks/BACKUP-RESTORE.md | 10 +++--- docs/runbooks/FIRST-SERVER-BOOTSTRAP.md | 48 ++++++++++++------------- 2 files changed, 29 insertions(+), 29 deletions(-) diff --git a/docs/runbooks/BACKUP-RESTORE.md b/docs/runbooks/BACKUP-RESTORE.md index d514c92..54097e6 100644 --- a/docs/runbooks/BACKUP-RESTORE.md +++ b/docs/runbooks/BACKUP-RESTORE.md @@ -13,7 +13,7 @@ Back up authoritative PostgreSQL data and the minimum manifest required to valid - Redis/AOF/cache/queue data - `.env` or secret files -- deployment SSH/registry/object-store credentials +- deployment SSH and object-store credentials - backup decryption key - TLS private keys or ACME account data - raw challenge CSV archives @@ -53,7 +53,7 @@ The manifest stores no password, connection URL, secret value, license identifie 1. Acquire a backup lock that conflicts with restore and destructive migration but need not block ordinary application reads/writes. 2. Resolve and verify the exact production host/database identity. Validate free disk for temporary output plus the encrypted artifact. -3. Confirm PostgreSQL health, client/server major compatibility, active release id, and migration version. +3. Confirm PostgreSQL health, client/server major compatibility, active three-hash release id, and migration version. 4. Create a consistent logical dump using PostgreSQL 18 `pg_dump` custom or directory format. Capture stderr and fail on a non-zero exit. 5. Inspect the archive table of contents with `pg_restore --list`; reject an empty/truncated/unreadable archive. 6. Build the non-secret manifest and calculate the plaintext archive SHA-256. @@ -86,7 +86,7 @@ Run monthly and before any approved destructive migration. 5. Treat the dump as trusted code only if it came from the approved bucket/path and passes provenance checks. PostgreSQL warns that restore executes statements controlled by the dump source. 6. Create a new empty database from `template0` or the equivalent approved empty-cluster procedure. Do not reuse a partially populated database. 7. Restore with error-stop semantics and reviewed owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets. -8. Run the selected release's validation suite: +8. Check out/build the selected release record if needed, then run its validation suite: - expected migration/import manifests - representative table/session/attempt counts and invariants - at least two merchant logins and cross-tenant denial @@ -115,10 +115,10 @@ If any input is absent or mismatched, exit before stopping writers, dropping/ren 1. Announce maintenance and stop public writes/new sessions. Preserve an operator-only health path. 2. Acquire the exclusive deployment/restore lock; stop worker before backend so no jobs write during restore. -3. Capture the active/previous release manifests, database identity, incident timestamps, and current-state backup when safe. +3. Capture the active/previous three-hash release records, database identity, incident timestamps, and current-state backup when safe. 4. Perform all download/decrypt/checksum/archive/provenance checks from the rehearsal procedure before mutating database state. 5. Restore into a newly created empty database/cluster. Do not pipe an unchecked dump directly into the active production database and do not recursively delete a computed volume/path. -6. Validate restore completeness and application invariants using the release compatible with the backup manifest. Apply only explicitly approved forward migrations. +6. Validate restore completeness and application invariants using the three recorded repository commits compatible with the backup manifest. Apply only explicitly approved forward migrations. 7. Start with empty Redis. Start backend/worker, confirm readiness, idempotent stale-run reconciliation, cold-cache analytics, sessions, and SSE behavior. 8. Switch application connectivity to the validated restored destination using the approved implementation strategy. 9. Run public TLS/login/dashboard plus private database/queue/tenant smoke checks. Observe for 15 minutes. diff --git a/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md b/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md index ce6c990..e1f49f2 100644 --- a/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md +++ b/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md @@ -16,10 +16,10 @@ Before connecting to the host, record in the change ticket/release record: - operator names, maintenance window, rollback/abandonment condition, and emergency contact - DNS zone owner and current `A`/`AAAA`/CAA records for `zarinpal.amiirkhl.ir` - SSH source allowlist and at least two tested operator public keys -- selected rootless or dedicated rootful Docker model (a remaining decision in the platform spec) -- private registry hostname and read-only production pull identity +- dedicated deployment username that will receive Docker-group access +- approved deployment, backend, and frontend repository URLs and expected nested paths - object-storage bucket/endpoint, retention capability, backup credentials, and separately escrowed encryption key -- approved release manifest containing exact image digests and all three repository SHAs +- target `main` branches and an empty location for the first three-hash release record - secret inventory and owner; never put values in the ticket, shell history, or this repository Do not change DNS before the host passes local and direct-IP validation. Do not reuse Qlockify credentials, certificates, volumes, backup paths, or deploy keys. @@ -58,23 +58,23 @@ Evidence: OS release/kernel, update result, time-sync state, two successful key ## Phase 3: install and constrain Docker 1. Install Docker Engine and the Compose plugin only from Docker's approved repository for the selected OS; pin/record installed versions. -2. Apply the approved rootless or dedicated rootful design. If the deploy user joins the `docker` group, record that it is effectively root-equivalent and restrict that account accordingly. +2. Add only the dedicated trusted deploy user to the `docker` group. Record that this access is effectively root-equivalent and restrict the account accordingly. 3. Configure the daemon's bounded logging policy before starting application containers. 4. Ensure Docker starts after reboot and that no unauthenticated TCP daemon socket is enabled. -5. Log in to the private registry using the production read-only pull identity. Store credentials in the approved host credential mechanism, not the deployment checkout or shell history. -6. Pull a harmless pinned test image, verify its digest, run it without published ports, remove the test container, and retain the image only if policy allows. +5. Pull a harmless pinned public test image, verify its tag/digest, run it without published ports, then remove the test container. +6. Confirm the deploy user can run Docker Compose without using a broader shared administrator account. -Evidence: Engine/Compose versions, daemon security configuration, socket ownership, log driver/options, registry identity scope, and digest verification. +Evidence: Engine/Compose versions, daemon security configuration, socket/group ownership, deploy-user membership, log driver/options, and test-image result. ## Phase 4: install the deployment control plane -1. Create a dedicated deploy machine identity with read access to the deployment repository and no write/push permission. The production host does not need backend/frontend source checkouts when deploying immutable images. -2. Clone the deployment repository into `/srv/zarinpal-analytics/current` or the approved release-control path. Pin/checkout the reviewed deployment SHA from the release manifest without destructive synchronization of unrelated paths. -3. Validate repository ownership, remote URL, current SHA, and clean status. +1. Create one server Git identity/key with read-only access to all three repositories and no write/push permission. +2. Clone the deployment repository into the approved deployment root, then clone backend and frontend into its exact ignored nested paths. +3. Validate ownership, remote URL, `main` branch, current SHA, clean status, and independent `.git` directory for every repository. 4. Materialize protected non-secret configuration and secret files outside the checkout. Set owner/mode before writing secret values; verify consumers receive only required files. -5. Install the production release manifest and verify its signature/checksum, exact target environment, previous release pointer, image digests, and configuration checksum. -6. Pull every manifest image by digest. Fail if a digest is missing, mutable, untrusted, or blocked by the vulnerability/provenance gate. -7. Render/validate Compose configuration in a way that does not print secret values into logs. Confirm only Nginx publishes 80/443 and operator endpoints bind only to loopback/private paths. +5. Fetch each repository and confirm it can fast-forward cleanly to `origin/main`; do not reset, build, or start yet. +6. Record the three initial commit hashes and selected infrastructure image tags in a non-secret draft release record. +7. Render/validate Compose configuration without printing secret values. Confirm only Nginx publishes 80/443. Do not run `docker compose up`, migrations, import, credential generation, restore, or DNS cutover during this phase. @@ -94,14 +94,14 @@ If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/c 1. Acquire the production deployment lock and record the operator/release id. 2. Start PostgreSQL and Redis only. Verify named-volume targets, authentication, resource limits, and health. -3. Run the one-off migration at the candidate backend digest. Confirm exit zero and the exact expected migration version. +3. Run the one-off migration from the recorded backend checkout/build. Confirm exit zero and the exact expected migration version. 4. Start backend and worker. Verify liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports. 5. Start frontend and verify the static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs. 6. Start/reload Nginx and run public generic health, canonical redirect, security-header, login-page, and authenticated smoke checks. -7. Verify Prometheus targets/dashboards and test notification delivery before relying on the 15-minute release watch. -8. Record the active release manifest only after all postflight checks pass. +7. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access. +8. Record the active three-hash release only after all postflight checks pass. -Failure before the active-manifest switch restores the prior/empty known state. Do not improvise a database down migration. +Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration. ## Phase 7: one-time data and demo setup @@ -121,22 +121,22 @@ Never add the source archive or credentials to Git, a container image, backup pa 1. Run the first encrypted logical PostgreSQL backup. 2. Verify archive listing/checksum, encrypted local artifact, remote upload, remote checksum, manifest, and retention counts. 3. Restore that backup to a clean non-production destination and complete application/tenant/analytics validation within the two-hour RTO. -4. Test certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age alerts. +4. Perform and record the MVP manual certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age checks. 5. Record current release, DNS, certificate, backup id, achieved RPO/RTO, remaining exceptions, and on-call ownership. 6. Remove temporary bootstrap credentials/files and close provider rules that are no longer needed. -The host is not considered production-ready until restore rehearsal and alert delivery pass. +The host is not considered MVP-ready until restore rehearsal and every manual operator check passes. Prometheus/Grafana and automated alert delivery are the required next phase before AI work. ## Stop/rollback conditions Stop without progressing when: - recovery console or second-operator SSH access is not proven -- resolved paths, host identity, environment, or release manifest do not match -- an image is not digest-pinned or provenance/vulnerability checks fail +- resolved paths, host identity, repository remotes/branches, environment, or release record do not match +- a repository is dirty/diverged unexpectedly or a build/source dependency cannot be verified - unexpected ports are externally reachable - DNS points somewhere unexpected or certificate names/chain are invalid -- migration, readiness, authenticated smoke, tenant isolation, worker/SSE, monitoring, backup, or restore rehearsal fails +- migration, readiness, authenticated smoke, tenant isolation, worker/SSE, manual operational checks, backup, or restore rehearsal fails - secrets appear in output, files have broad permissions, or the deployment checkout is dirty unexpectedly Rollback DNS to the recorded prior values when the new host cannot pass the cutover window. Preserve logs and manifests, revoke any exposed credential, and do not delete the failed host until evidence and required backups are secured. @@ -147,11 +147,11 @@ Rollback DNS to the recorded prior values when the new host cannot pass the cuto - [ ] Console plus two key-only operator paths proven - [ ] Firewall external tests show only approved ports - [ ] Docker/Compose versions and privilege model recorded -- [ ] Deployment SHA and release manifest verified +- [ ] Three repository remotes/branches/SHAs and release record verified - [ ] Secrets outside Git with restrictive ownership/mode - [ ] TLS issuance and renewal tested - [ ] Migration, API, worker, frontend/font, Nginx, SSE, and tenant smoke pass - [ ] One-time import reconciles and demo credentials stay outside logs/Git -- [ ] Monitoring targets and alert delivery pass +- [ ] MVP manual health/log/disk/certificate/backup/queue checks pass - [ ] Encrypted off-host backup and clean restore rehearsal pass within RTO - [ ] Active/previous release ids and operator handoff recorded