diff --git a/docs/runbooks/BACKUP-RESTORE.md b/docs/runbooks/BACKUP-RESTORE.md new file mode 100644 index 0000000..d514c92 --- /dev/null +++ b/docs/runbooks/BACKUP-RESTORE.md @@ -0,0 +1,154 @@ +# PostgreSQL backup and restore runbook + +- Status: Draft operational contract; scripts not yet implemented +- RPO: 24 hours +- RTO: two hours +- Last updated: 2026-08-20 + +This runbook adapts Qlockify's logical dump, encrypted S3-compatible upload, local/remote retention, and restore flow while removing environment files from the backup and adding exact-destination/production confirmation guards. + +## Scope and exclusions + +Back up authoritative PostgreSQL data and the minimum manifest required to validate it. Do not include: + +- Redis/AOF/cache/queue data +- `.env` or secret files +- deployment SSH/registry/object-store credentials +- backup decryption key +- TLS private keys or ACME account data +- raw challenge CSV archives +- demo plaintext credential exports +- frontend font/license files +- Prometheus/Grafana runtime volumes or container logs + +Redis loss is recovered through empty Redis plus idempotent analytics reconciliation/recomputation, not restore. + +## Backup package contract + +Each encrypted artifact corresponds to one manifest with: + +```text +format_version +backup_id +created_at_utc +completed_at_utc +source_environment +source_host_id +postgres_server_version +dump_client_version +database_name +database_archive_sha256 +database_archive_bytes +schema_migration_version +active_release_id +deployment_sha +backend_sha +frontend_sha +encryption_scheme_version +``` + +The manifest stores no password, connection URL, secret value, license identifier/text, raw font copy, CSV path/content, or merchant data sample. A checksum of the encrypted remote object is recorded separately from the plaintext archive checksum. + +## Scheduled backup procedure + +1. Acquire a backup lock that conflicts with restore and destructive migration but need not block ordinary application reads/writes. +2. Resolve and verify the exact production host/database identity. Validate free disk for temporary output plus the encrypted artifact. +3. Confirm PostgreSQL health, client/server major compatibility, active release id, and migration version. +4. Create a consistent logical dump using PostgreSQL 18 `pg_dump` custom or directory format. Capture stderr and fail on a non-zero exit. +5. Inspect the archive table of contents with `pg_restore --list`; reject an empty/truncated/unreadable archive. +6. Build the non-secret manifest and calculate the plaintext archive SHA-256. +7. Encrypt the package before moving it into durable local retention. The approved implementation uses authenticated encryption/KDF parameters recorded by `encryption_scheme_version`. +8. Upload over TLS to the private S3-compatible destination. Upload to a temporary object key, verify remote size/checksum/metadata, then finalize the immutable timestamped key. +9. Mark the backup successful only after remote verification. Export success timestamp/duration/size metrics. +10. Retain the newest three verified encrypted local artifacts and seven verified encrypted remote artifacts. Delete stale artifacts only after the new remote object is verified; never delete the last known-good local or remote backup. +11. Release the lock and ensure temporary plaintext files are removed from the exact temporary directory. + +A cleanup failure is a ticket, not a reason to mark the backup failed if the verified new copy exists. An upload/verification failure is a backup failure and preserves all prior copies. + +## Daily verification + +- last verified remote backup age is under 30 hours +- archive/manifest/encrypted-object checksums and sizes are non-zero and agree +- retention counts are at least one and no greater than approved limits after cleanup +- backup duration/size are within expected bounds or investigated +- no excluded filenames/classes appear in the archive listing +- local backup directory and remote bucket are not publicly readable +- decryption key is retrievable from separate escrow by authorized operators without exposing it in the check record + +## Clean-destination restore rehearsal + +Run monthly and before any approved destructive migration. + +1. Select an explicit backup id, not an unexamined implicit `latest`, and record why it was chosen. +2. Provision an empty non-production destination with compatible PostgreSQL and enough free space. Confirm its environment/hostname/database do not match production. +3. Download the exact encrypted object to a new restrictive temporary directory. +4. Verify remote-object checksum/size, decrypt, validate package and archive checksum, parse the manifest, and inspect `pg_restore --list`. +5. Treat the dump as trusted code only if it came from the approved bucket/path and passes provenance checks. PostgreSQL warns that restore executes statements controlled by the dump source. +6. Create a new empty database from `template0` or the equivalent approved empty-cluster procedure. Do not reuse a partially populated database. +7. Restore with error-stop semantics and reviewed owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets. +8. Run the selected release's validation suite: + - expected migration/import manifests + - representative table/session/attempt counts and invariants + - at least two merchant logins and cross-tenant denial + - payer-card keys absent from API/log output + - Redis initially empty, analytics recomputation succeeds, four sections/SSE work + - frontend and API compatibility for the manifest's release +9. Record elapsed time, achieved RPO, achieved RTO, sizes/counts, validation result, operator, and remediation. +10. Destroy rehearsal resources only after evidence is retained and the exact destination is revalidated. + +## Production restore approval guard + +Production restore is destructive/exclusive and always requires explicit human approval. The future implementation must require all of these independent inputs: + +- `RESTORE_ENVIRONMENT=production` +- exact expected host identity matching the current provider instance +- exact expected database name +- explicit backup id/object key (no automatic `latest`) +- a typed confirmation phrase containing both production and the backup id +- active maintenance mode and deployment/restore lock +- approved incident/change record and named operator +- current-state backup when PostgreSQL is readable and doing so will not worsen the incident + +If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `pg_restore`. + +## Production restore sequence + +1. Announce maintenance and stop public writes/new sessions. Preserve an operator-only health path. +2. Acquire the exclusive deployment/restore lock; stop worker before backend so no jobs write during restore. +3. Capture the active/previous release manifests, database identity, incident timestamps, and current-state backup when safe. +4. Perform all download/decrypt/checksum/archive/provenance checks from the rehearsal procedure before mutating database state. +5. Restore into a newly created empty database/cluster. Do not pipe an unchecked dump directly into the active production database and do not recursively delete a computed volume/path. +6. Validate restore completeness and application invariants using the release compatible with the backup manifest. Apply only explicitly approved forward migrations. +7. Start with empty Redis. Start backend/worker, confirm readiness, idempotent stale-run reconciliation, cold-cache analytics, sessions, and SSE behavior. +8. Switch application connectivity to the validated restored destination using the approved implementation strategy. +9. Run public TLS/login/dashboard plus private database/queue/tenant smoke checks. Observe for 15 minutes. +10. Reopen traffic, mark the restore complete, and schedule a fresh backup. Retain the old database until the incident owner approves disposal under a separate exact-target procedure. + +Do not automatically restore `.env`/secrets, TLS, Redis, or frontend assets from a database backup. Reconstruct platform configuration from Git/release records and secrets from separate escrow. + +## Failure handling + +| Failure | Action | +|---|---| +| Backup lock already held | Exit without starting a second backup; alert if stale | +| `pg_dump` non-zero or unreadable listing | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold | +| Encryption failure | Do not persist/upload plaintext; secure exact temp path and investigate | +| Remote upload/checksum failure | Keep encrypted local artifact and all prior remote copies; bounded retry; alert | +| Retention cleanup failure | Keep extra copies; ticket; never broaden deletion globs/paths | +| Manifest/archive mismatch | Quarantine artifact and select another known-good backup | +| Restore SQL/object failure | Keep traffic closed, preserve logs/destination, do not rerun blindly into partial DB | +| Post-restore invariant failure | Do not switch/reopen; return to old database if safe or select another backup | +| Missing decryption key | Stop; encrypted backup is unrecoverable without the separately escrowed key | +| Wrong environment/host/database confirmation | Abort before mutation and record denied attempt | + +## Acceptance evidence + +- [ ] Daily backup completed and verified remotely within 30 hours +- [ ] Backup contains PostgreSQL archive plus non-secret manifest only +- [ ] Three local/seven remote verified encrypted artifacts retained +- [ ] Restore rehearsal uses an explicit backup id and clean non-production database +- [ ] Archive trust/checksum/listing/compatibility checks precede restore +- [ ] Application, tenant, session, cold-cache, worker, and SSE validations pass +- [ ] Achieved RPO <=24 hours and RTO <=2 hours recorded +- [ ] Every missing production confirmation factor aborts before mutation +- [ ] No Redis, env, secrets, TLS keys, CSV, credentials, font/license, logs, or monitoring runtime data enter backup diff --git a/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md b/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md new file mode 100644 index 0000000..ce6c990 --- /dev/null +++ b/docs/runbooks/FIRST-SERVER-BOOTSTRAP.md @@ -0,0 +1,157 @@ +# First-server bootstrap runbook + +- Status: Draft operational procedure; commands/artifacts not yet implemented +- Target: `zarinpal.amiirkhl.ir` +- Owner: Platform operator +- Last updated: 2026-08-20 + +This is the safe order for preparing a new production host. It is intentionally a human-reviewed checklist, not a copy-paste bootstrap script. Stop on any failed validation and preserve the existing server/DNS state until the failure is understood. + +## Preconditions and recorded inputs + +Before connecting to the host, record in the change ticket/release record: + +- server provider, instance id, public IPv4/IPv6, region, CPU, RAM, disk type/size, and console/recovery access +- approved OS image and checksum/source; initial target is a supported Ubuntu LTS minimal image +- operator names, maintenance window, rollback/abandonment condition, and emergency contact +- DNS zone owner and current `A`/`AAAA`/CAA records for `zarinpal.amiirkhl.ir` +- SSH source allowlist and at least two tested operator public keys +- selected rootless or dedicated rootful Docker model (a remaining decision in the platform spec) +- private registry hostname and read-only production pull identity +- object-storage bucket/endpoint, retention capability, backup credentials, and separately escrowed encryption key +- approved release manifest containing exact image digests and all three repository SHAs +- secret inventory and owner; never put values in the ticket, shell history, or this repository + +Do not change DNS before the host passes local and direct-IP validation. Do not reuse Qlockify credentials, certificates, volumes, backup paths, or deploy keys. + +## Phase 1: establish recoverable access + +1. Open the provider console/recovery channel and confirm it works independently of SSH. +2. Log in using the provider's initial method. Create named operator accounts and install only approved public keys. +3. Confirm a second operator can log in in a new session before changing SSH configuration. +4. Disable direct root SSH and password authentication only after key access and recovery console are proven. +5. Restrict SSH to the operator source allowlist at the provider firewall and host firewall. Keep the current session open while testing a new one. +6. Configure time synchronization and verify UTC plus the documented `Asia/Tehran` application assumption. The host itself should log in UTC. +7. Apply all supported security updates and reboot if required. Re-test console and SSH access after reboot. + +Evidence: OS release/kernel, update result, time-sync state, two successful key logins, failed password/root-login test, and firewall rules. Do not record public key private material or session output containing secrets. + +## Phase 2: harden and size the host + +1. Set a unique hostname and record the provider instance id alongside it. +2. Configure default-deny inbound firewall rules. Initially allow SSH from the operator allowlist and TCP 80/443; verify from both an allowed and disallowed external source. +3. Enable unattended security updates with a documented reboot policy and alert path. +4. Configure disk/inode monitoring, time sync, audit/system logs, log rotation, and OOM visibility before application deployment. +5. Create dedicated paths with explicit ownership: + + ```text + /srv/zarinpal-analytics/ deployment control plane and release records + /etc/zarinpal-analytics/config/ protected non-secret environment configuration + /etc/zarinpal-analytics/secrets/ runtime secret files, mode 0600 or stricter + /var/lib/zarinpal-analytics/ host-managed operational state if required + /var/backups/zarinpal-analytics/ encrypted local backup retention only + ``` + +6. Verify every resolved absolute path is on the intended filesystem, not a symlink to a broad/unexpected target, before applying ownership or any future recursive operation. +7. Reserve enough free space for PostgreSQL growth, one temporary logical dump, three encrypted local backups, container images, and logs while retaining the platform alert floor of at least 20% and 10 GiB free. + +## Phase 3: install and constrain Docker + +1. Install Docker Engine and the Compose plugin only from Docker's approved repository for the selected OS; pin/record installed versions. +2. Apply the approved rootless or dedicated rootful design. If the deploy user joins the `docker` group, record that it is effectively root-equivalent and restrict that account accordingly. +3. Configure the daemon's bounded logging policy before starting application containers. +4. Ensure Docker starts after reboot and that no unauthenticated TCP daemon socket is enabled. +5. Log in to the private registry using the production read-only pull identity. Store credentials in the approved host credential mechanism, not the deployment checkout or shell history. +6. Pull a harmless pinned test image, verify its digest, run it without published ports, remove the test container, and retain the image only if policy allows. + +Evidence: Engine/Compose versions, daemon security configuration, socket ownership, log driver/options, registry identity scope, and digest verification. + +## Phase 4: install the deployment control plane + +1. Create a dedicated deploy machine identity with read access to the deployment repository and no write/push permission. The production host does not need backend/frontend source checkouts when deploying immutable images. +2. Clone the deployment repository into `/srv/zarinpal-analytics/current` or the approved release-control path. Pin/checkout the reviewed deployment SHA from the release manifest without destructive synchronization of unrelated paths. +3. Validate repository ownership, remote URL, current SHA, and clean status. +4. Materialize protected non-secret configuration and secret files outside the checkout. Set owner/mode before writing secret values; verify consumers receive only required files. +5. Install the production release manifest and verify its signature/checksum, exact target environment, previous release pointer, image digests, and configuration checksum. +6. Pull every manifest image by digest. Fail if a digest is missing, mutable, untrusted, or blocked by the vulnerability/provenance gate. +7. Render/validate Compose configuration in a way that does not print secret values into logs. Confirm only Nginx publishes 80/443 and operator endpoints bind only to loopback/private paths. + +Do not run `docker compose up`, migrations, import, credential generation, restore, or DNS cutover during this phase. + +## Phase 5: DNS and TLS staging + +1. Lower DNS TTL in advance if an existing record is moving. Preserve the previous values and rollback time. +2. Point `A`/`AAAA` only after direct-IP Nginx/host validation succeeds. Remove an `AAAA` record if IPv6 is not actually routed and firewalled correctly. +3. Verify public authoritative DNS from multiple resolvers and confirm the address matches the current server. +4. Start only the minimum edge services required for ACME HTTP-01. Verify the challenge path from an external network. +5. Obtain a staging certificate first and exercise renewal. Then request the production certificate. +6. Validate certificate names, chain, expiry, TLS 1.2/1.3, canonical redirects, unknown-host rejection, and renewal permissions. +7. Enable HSTS only after successful production issuance and renewal evidence; preload remains a separate approval. + +If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/challenge routing, and continue serving no plaintext application content. + +## Phase 6: first database and application release + +1. Acquire the production deployment lock and record the operator/release id. +2. Start PostgreSQL and Redis only. Verify named-volume targets, authentication, resource limits, and health. +3. Run the one-off migration at the candidate backend digest. Confirm exit zero and the exact expected migration version. +4. Start backend and worker. Verify liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports. +5. Start frontend and verify the static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs. +6. Start/reload Nginx and run public generic health, canonical redirect, security-header, login-page, and authenticated smoke checks. +7. Verify Prometheus targets/dashboards and test notification delivery before relying on the 15-minute release watch. +8. Record the active release manifest only after all postflight checks pass. + +Failure before the active-manifest switch restores the prior/empty known state. Do not improvise a database down migration. + +## Phase 7: one-time data and demo setup + +This phase is deliberately separate from server/application bootstrap. + +1. Copy or mount the approved challenge archive to an absolute operator-owned path outside Git. Validate path, owner, permissions, size, and SHA-256. +2. Confirm no migration, restore, backup, or import is active. Acquire the import lock. +3. Invoke the explicit one-off import service with a read-only mount of exactly that file. +4. Review import status, checksum, source/accepted/rejected counts, quality report, and reconciliation before declaring the dataset ready. +5. Run demo credential generation as a separate explicit administrative job. Select a secure one-time output destination outside the repository and Compose logs. +6. Test at least two merchant accounts and cross-tenant isolation. Remove or escrow plaintext credential output according to the approved operator procedure. + +Never add the source archive or credentials to Git, a container image, backup package, or diagnostic bundle. + +## Phase 8: backup and handoff gate + +1. Run the first encrypted logical PostgreSQL backup. +2. Verify archive listing/checksum, encrypted local artifact, remote upload, remote checksum, manifest, and retention counts. +3. Restore that backup to a clean non-production destination and complete application/tenant/analytics validation within the two-hour RTO. +4. Test certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age alerts. +5. Record current release, DNS, certificate, backup id, achieved RPO/RTO, remaining exceptions, and on-call ownership. +6. Remove temporary bootstrap credentials/files and close provider rules that are no longer needed. + +The host is not considered production-ready until restore rehearsal and alert delivery pass. + +## Stop/rollback conditions + +Stop without progressing when: + +- recovery console or second-operator SSH access is not proven +- resolved paths, host identity, environment, or release manifest do not match +- an image is not digest-pinned or provenance/vulnerability checks fail +- unexpected ports are externally reachable +- DNS points somewhere unexpected or certificate names/chain are invalid +- migration, readiness, authenticated smoke, tenant isolation, worker/SSE, monitoring, backup, or restore rehearsal fails +- secrets appear in output, files have broad permissions, or the deployment checkout is dirty unexpectedly + +Rollback DNS to the recorded prior values when the new host cannot pass the cutover window. Preserve logs and manifests, revoke any exposed credential, and do not delete the failed host until evidence and required backups are secured. + +## Completion evidence + +- [ ] Provider/host identity and capacity recorded +- [ ] Console plus two key-only operator paths proven +- [ ] Firewall external tests show only approved ports +- [ ] Docker/Compose versions and privilege model recorded +- [ ] Deployment SHA and release manifest verified +- [ ] Secrets outside Git with restrictive ownership/mode +- [ ] TLS issuance and renewal tested +- [ ] Migration, API, worker, frontend/font, Nginx, SSE, and tenant smoke pass +- [ ] One-time import reconciles and demo credentials stay outside logs/Git +- [ ] Monitoring targets and alert delivery pass +- [ ] Encrypted off-host backup and clean restore rehearsal pass within RTO +- [ ] Active/previous release ids and operator handoff recorded