docs: approve platform operations runbooks

docs/runbooks/BACKUP-RESTORE.md

@@ -1,6 +1,6 @@
 # PostgreSQL backup and restore runbook
 
-- Status: Draft operational contract; scripts not yet implemented
+- Status: Approved procedure specification; scripts not yet implemented
 - RPO: 24 hours
 - RTO: two hours
 - Last updated: 2026-08-20
@@ -51,15 +51,17 @@ The manifest stores no password, connection URL, secret value, license identifie
 
 ## Scheduled backup procedure
 
+Run daily at `02:00 UTC` from the `ubuntu` user's reviewed host cron entry.
+
 1. Acquire a backup lock that conflicts with restore and destructive migration but need not block ordinary application reads/writes.
 2. Resolve and verify the exact production host/database identity. Validate free disk for temporary output plus the encrypted artifact.
 3. Confirm PostgreSQL health, client/server major compatibility, active three-hash release id, and migration version.
-4. Create a consistent logical dump using PostgreSQL 18 `pg_dump` custom or directory format. Capture stderr and fail on a non-zero exit.
+4. Create a consistent PostgreSQL 18 plain-SQL dump using Qlockify's reviewed `pg_dump --format=plain --clean --if-exists --no-owner --no-privileges` pattern. Capture stderr and fail on a non-zero exit.
-5. Inspect the archive table of contents with `pg_restore --list`; reject an empty/truncated/unreadable archive.
+5. Reject an empty output, verify the expected PostgreSQL dump header/format, and calculate its SHA-256.
 6. Build the non-secret manifest and calculate the plaintext archive SHA-256.
-7. Encrypt the package before moving it into durable local retention. The approved implementation uses authenticated encryption/KDF parameters recorded by `encryption_scheme_version`.
+7. Encrypt the package before durable local retention with Qlockify's `openssl enc -aes-256-cbc -salt -pbkdf2 -iter 200000` flow. The manifest records the scheme/version and SHA-256 integrity evidence because CBC is not authenticated encryption.
 8. Upload over TLS to the private S3-compatible destination. Upload to a temporary object key, verify remote size/checksum/metadata, then finalize the immutable timestamped key.
-9. Mark the backup successful only after remote verification. Export success timestamp/duration/size metrics.
+9. Mark the backup successful only after remote verification. Record success timestamp, duration, and size in structured MVP evidence; export the same signals as metrics in the required post-MVP observability phase.
 10. Retain the newest three verified encrypted local artifacts and seven verified encrypted remote artifacts. Delete stale artifacts only after the new remote object is verified; never delete the last known-good local or remote backup.
 11. Release the lock and ensure temporary plaintext files are removed from the exact temporary directory.
 
@@ -71,7 +73,7 @@ A cleanup failure is a ticket, not a reason to mark the backup failed if the ver
 - archive/manifest/encrypted-object checksums and sizes are non-zero and agree
 - retention counts are at least one and no greater than approved limits after cleanup
 - backup duration/size are within expected bounds or investigated
-- no excluded filenames/classes appear in the archive listing
+- no excluded filenames/classes appear in the package-content inspection
 - local backup directory and remote bucket are not publicly readable
 - decryption key is retrievable from separate escrow by authorized operators without exposing it in the check record
 
@@ -82,10 +84,10 @@ Run monthly and before any approved destructive migration.
 1. Select an explicit backup id, not an unexamined implicit `latest`, and record why it was chosen.
 2. Provision an empty non-production destination with compatible PostgreSQL and enough free space. Confirm its environment/hostname/database do not match production.
 3. Download the exact encrypted object to a new restrictive temporary directory.
-4. Verify remote-object checksum/size, decrypt, validate package and archive checksum, parse the manifest, and inspect `pg_restore --list`.
+4. Verify remote-object checksum/size, decrypt, validate the package and plain-SQL checksum/header, and parse the manifest.
 5. Treat the dump as trusted code only if it came from the approved bucket/path and passes provenance checks. PostgreSQL warns that restore executes statements controlled by the dump source.
 6. Create a new empty database from `template0` or the equivalent approved empty-cluster procedure. Do not reuse a partially populated database.
-7. Restore with error-stop semantics and reviewed owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets.
+7. Restore with `psql -X --set ON_ERROR_STOP=on` and the reviewed transaction/owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets.
 8. Check out/build the selected release record if needed, then run its validation suite:
    - expected migration/import manifests
    - representative table/session/attempt counts and invariants
@@ -109,7 +111,7 @@ Production restore is destructive/exclusive and always requires explicit human a
 - approved incident/change record and named operator
 - current-state backup when PostgreSQL is readable and doing so will not worsen the incident
 
-If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `pg_restore`.
+If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `psql`.
 
 ## Production restore sequence
 
@@ -131,7 +133,7 @@ Do not automatically restore `.env`/secrets, TLS, Redis, or frontend assets from
 | Failure | Action |
 |---|---|
 | Backup lock already held | Exit without starting a second backup; alert if stale |
-| `pg_dump` non-zero or unreadable listing | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold |
+| `pg_dump` non-zero, empty, or invalid plain-SQL output | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold |
 | Encryption failure | Do not persist/upload plaintext; secure exact temp path and investigate |
 | Remote upload/checksum failure | Keep encrypted local artifact and all prior remote copies; bounded retry; alert |
 | Retention cleanup failure | Keep extra copies; ticket; never broaden deletion globs/paths |

docs/runbooks/FIRST-SERVER-BOOTSTRAP.md

@@ -1,6 +1,6 @@
 # First-server bootstrap runbook
 
-- Status: Draft operational procedure; commands/artifacts not yet implemented
+- Status: Approved procedure specification; commands/artifacts not yet implemented
 - Target: `zarinpal.amiirkhl.ir`
 - Owner: Platform operator
 - Last updated: 2026-08-20
@@ -12,17 +12,17 @@ This is the safe order for preparing a new production host. It is intentionally
 Before connecting to the host, record in the change ticket/release record:
 
 - server provider, instance id, public IPv4/IPv6, region, CPU, RAM, disk type/size, and console/recovery access
-- approved OS image and checksum/source; initial target is a supported Ubuntu LTS minimal image
+- Ubuntu 24.04 LTS image/source for the new 2-core/2-GB server, with Docker already installed and the `ubuntu` user available
 - operator names, maintenance window, rollback/abandonment condition, and emergency contact
 - DNS zone owner and current `A`/`AAAA`/CAA records for `zarinpal.amiirkhl.ir`
 - SSH source allowlist and at least two tested operator public keys
-- dedicated deployment username that will receive Docker-group access
+- confirmation that the trusted `ubuntu` user has the required Docker-group access
 - approved deployment, backend, and frontend repository URLs and expected nested paths
 - object-storage bucket/endpoint, retention capability, backup credentials, and separately escrowed encryption key
 - target `main` branches and an empty location for the first three-hash release record
 - secret inventory and owner; never put values in the ticket, shell history, or this repository
 
-Do not change DNS before the host passes local and direct-IP validation. Do not reuse Qlockify credentials, certificates, volumes, backup paths, or deploy keys.
+Do not change DNS before the host passes local and direct-IP validation. Do not reuse Qlockify credentials, certificates, volumes, backup paths, or Git credentials.
 
 ## Phase 1: establish recoverable access
 
@@ -42,37 +42,38 @@ Evidence: OS release/kernel, update result, time-sync state, two successful key
 2. Configure default-deny inbound firewall rules. Initially allow SSH from the operator allowlist and TCP 80/443; verify from both an allowed and disallowed external source.
 3. Enable unattended security updates with a documented reboot policy and alert path.
 4. Configure disk/inode monitoring, time sync, audit/system logs, log rotation, and OOM visibility before application deployment.
-5. Create dedicated paths with explicit ownership:
+5. Create the approved Qlockify-style runtime paths with explicit `ubuntu` ownership:
 
    ```text
-   /srv/zarinpal-analytics/           deployment control plane and release records
+   /home/ubuntu/zarinpal-analytics-deployment/           deployment repository and root .env
-   /etc/zarinpal-analytics/config/    protected non-secret environment configuration
+   /home/ubuntu/zarinpal-analytics-deployment/backend/   ignored backend checkout and backend .env
-   /etc/zarinpal-analytics/secrets/   runtime secret files, mode 0600 or stricter
+   /home/ubuntu/zarinpal-analytics-deployment/frontend/  ignored frontend checkout and frontend .env
-   /var/lib/zarinpal-analytics/       host-managed operational state if required
+   /home/ubuntu/zarinpal-analytics-deployment/releases/  ignored non-secret release records
-   /var/backups/zarinpal-analytics/   encrypted local backup retention only
+   /home/ubuntu/zarinpal-analytics-deployment/backups/   ignored encrypted local backup retention
    ```
 
 6. Verify every resolved absolute path is on the intended filesystem, not a symlink to a broad/unexpected target, before applying ownership or any future recursive operation.
 7. Reserve enough free space for PostgreSQL growth, one temporary logical dump, three encrypted local backups, container images, and logs while retaining the platform alert floor of at least 20% and 10 GiB free.
+8. Create and enable the approved 2 GB swap file, use restrictive permissions, persist it in `/etc/fstab`, and verify it after reboot.
 
 ## Phase 3: install and constrain Docker
 
-1. Install Docker Engine and the Compose plugin only from Docker's approved repository for the selected OS; pin/record installed versions.
+1. Verify the preinstalled Docker Engine and Compose plugin came from Docker's approved Ubuntu repository; update/install only if verification fails, then record installed versions.
-2. Add only the dedicated trusted deploy user to the `docker` group. Record that this access is effectively root-equivalent and restrict the account accordingly.
+2. Confirm only the trusted `ubuntu` user has the required `docker` group access. Record that this access is effectively root-equivalent and restrict the account accordingly.
 3. Configure the daemon's bounded logging policy before starting application containers.
 4. Ensure Docker starts after reboot and that no unauthenticated TCP daemon socket is enabled.
 5. Pull a harmless pinned public test image, verify its tag/digest, run it without published ports, then remove the test container.
-6. Confirm the deploy user can run Docker Compose without using a broader shared administrator account.
+6. Confirm the `ubuntu` user can run Docker Compose without using a broader shared administrator account.
 
 Evidence: Engine/Compose versions, daemon security configuration, socket/group ownership, deploy-user membership, log driver/options, and test-image result.
 
 ## Phase 4: install the deployment control plane
 
-1. Create one server Git identity/key with read-only access to all three repositories and no write/push permission.
+1. From the `ubuntu` account, clone the deployment repository into `/home/ubuntu/zarinpal-analytics-deployment`, then clone backend and frontend into its exact ignored nested paths. Enter the source-forge HTTPS username/password interactively; do not configure a credential helper, token, or deploy key.
-2. Clone the deployment repository into the approved deployment root, then clone backend and frontend into its exact ignored nested paths.
+2. Confirm no Git credential was persisted in repository configuration, shell scripts, environment files, or the operating-system credential store.
 3. Validate ownership, remote URL, `main` branch, current SHA, clean status, and independent `.git` directory for every repository.
-4. Materialize protected non-secret configuration and secret files outside the checkout. Set owner/mode before writing secret values; verify consumers receive only required files.
+4. Create the Qlockify-style Git-ignored environment files: deployment-root `.env` plus nested backend/frontend `.env`, all owned by `ubuntu`, mode 0600, and excluded from backup. Verify consumers receive only required values.
-5. Fetch each repository and confirm it can fast-forward cleanly to `origin/main`; do not reset, build, or start yet.
+5. Manually run `git pull --ff-only origin main` in each repository, entering HTTPS credentials interactively, and confirm each checkout equals `origin/main`; do not reset, build, or start yet.
 6. Record the three initial commit hashes and selected infrastructure image tags in a non-secret draft release record.
 7. Render/validate Compose configuration without printing secret values. Confirm only Nginx publishes 80/443.
 
@@ -81,7 +82,7 @@ Do not run `docker compose up`, migrations, import, credential generation, resto
 ## Phase 5: DNS and TLS staging
 
 1. Lower DNS TTL in advance if an existing record is moving. Preserve the previous values and rollback time.
-2. Point `A`/`AAAA` only after direct-IP Nginx/host validation succeeds. Remove an `AAAA` record if IPv6 is not actually routed and firewalled correctly.
+2. Point `A` (and `AAAA` only if IPv6 is actually routed/firewalled) for `zarinpal.amiirkhl.ir` after direct-IP Nginx/host validation. No `www` or alternate hostname is configured.
 3. Verify public authoritative DNS from multiple resolvers and confirm the address matches the current server.
 4. Start only the minimum edge services required for ACME HTTP-01. Verify the challenge path from an external network.
 5. Obtain a staging certificate first and exercise renewal. Then request the production certificate.
@@ -90,36 +91,35 @@ Do not run `docker compose up`, migrations, import, credential generation, resto
 
 If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/challenge routing, and continue serving no plaintext application content.
 
-## Phase 6: first database and application release
+## Phase 6: first database, dataset, and application release
 
 1. Acquire the production deployment lock and record the operator/release id.
-2. Start PostgreSQL and Redis only. Verify named-volume targets, authentication, resource limits, and health.
+2. Place the approved `challenge_data.csv.gz` at the exact Git-ignored deployment `data/incoming/` path. Verify regular-file/read-only properties, size, and pinned SHA-256; do not copy it into an image.
-3. Run the one-off migration from the recorded backend checkout/build. Confirm exit zero and the exact expected migration version.
+3. Invoke tracked `scripts/first-start.sh`. It starts PostgreSQL and Redis, waits with bounded backoff, applies the one-off migration, runs the one-shot importer with a read-only mount, and reconciles the pinned counts before starting public services.
-4. Start backend and worker. Verify liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
+4. Review the JSON/Markdown import report and durable import-run/issues metadata. Fatal validation, resource, migration, or reconciliation failure must exit nonzero before backend/frontend/Nginx activation.
-5. Start frontend and verify the static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
+5. Verify backend and worker liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
-6. Start/reload Nginx and run public generic health, canonical redirect, security-header, login-page, and authenticated smoke checks.
+6. Verify the frontend static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
-7. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
+7. Verify Nginx public generic health, canonical redirect, security headers, login page, SSE behavior, and that only 80/443 are public.
-8. Record the active three-hash release only after all postflight checks pass.
+8. Run demo credential generation as a separate explicit administrative job. Store one-time plaintext output outside Git/logs/backups.
+9. Authenticate as at least two merchant accounts, verify different dashboards and cross-tenant isolation, then remove or escrow plaintext credential output.
+10. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
+11. Record the active three-hash release only after all postflight checks pass.
 
 Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration.
 
-## Phase 7: one-time data and demo setup
+## Phase 7: bootstrap evidence review
 
-This phase is deliberately separate from server/application bootstrap.
+1. Confirm the sole completed import run matches the pinned checksum, schema, source/session/attempt/merchant/terminal/category counts, and contains no fatal issue.
-
+2. Confirm no raw staging remains and the ordinary `docker compose up -d` path cannot invoke migration/import/admin profiles.
-1. Copy or mount the approved challenge archive to an absolute operator-owned path outside Git. Validate path, owner, permissions, size, and SHA-256.
+3. Run `first-start.sh` a second time only in the controlled acceptance rehearsal and confirm the identical dataset is a safe no-op rather than another promotion.
-2. Confirm no migration, restore, backup, or import is active. Acquire the import lock.
+4. Archive only safe non-sensitive release/import evidence; exclude the CSV, raw rows, reports containing restricted values, and credentials.
-3. Invoke the explicit one-off import service with a read-only mount of exactly that file.
-4. Review import status, checksum, source/accepted/rejected counts, quality report, and reconciliation before declaring the dataset ready.
-5. Run demo credential generation as a separate explicit administrative job. Select a secure one-time output destination outside the repository and Compose logs.
-6. Test at least two merchant accounts and cross-tenant isolation. Remove or escrow plaintext credential output according to the approved operator procedure.
 
 Never add the source archive or credentials to Git, a container image, backup package, or diagnostic bundle.
 
 ## Phase 8: backup and handoff gate
 
 1. Run the first encrypted logical PostgreSQL backup.
-2. Verify archive listing/checksum, encrypted local artifact, remote upload, remote checksum, manifest, and retention counts.
+2. Verify package contents, plain-SQL header/checksum, encrypted local artifact, remote upload, remote checksum, manifest, and retention counts.
 3. Restore that backup to a clean non-production destination and complete application/tenant/analytics validation within the two-hour RTO.
 4. Perform and record the MVP manual certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age checks.
 5. Record current release, DNS, certificate, backup id, achieved RPO/RTO, remaining exceptions, and on-call ownership.