starcoach-hackathon/zarinpal-analytics-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: approve platform operations runbooks

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 18:54:59 +03:30
parent e93eca1384
commit 83d2e13d14
2 changed files with 48 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/runbooks/BACKUP-RESTORE.md
View File

@@ -1,6 +1,6 @@
# PostgreSQL backup and restore runbook
- Status: Draft operational contract; scripts not yet implemented
- Status: Approved procedure specification; scripts not yet implemented
- RPO: 24 hours
- RTO: two hours
- Last updated: 2026-08-20
@@ -51,15 +51,17 @@ The manifest stores no password, connection URL, secret value, license identifie
## Scheduled backup procedure
Run daily at `02:00 UTC` from the `ubuntu` user's reviewed host cron entry.
1. Acquire a backup lock that conflicts with restore and destructive migration but need not block ordinary application reads/writes.
2. Resolve and verify the exact production host/database identity. Validate free disk for temporary output plus the encrypted artifact.
3. Confirm PostgreSQL health, client/server major compatibility, active three-hash release id, and migration version.
4. Create a consistent logical dump using PostgreSQL 18 `pg_dump` custom or directory format. Capture stderr and fail on a non-zero exit.
5. Inspect the archive table of contents with `pg_restore --list`; reject an empty/truncated/unreadable archive.
4. Create a consistent PostgreSQL 18 plain-SQL dump using Qlockify's reviewed `pg_dump --format=plain --clean --if-exists --no-owner --no-privileges` pattern. Capture stderr and fail on a non-zero exit.
5. Reject an empty output, verify the expected PostgreSQL dump header/format, and calculate its SHA-256.
6. Build the non-secret manifest and calculate the plaintext archive SHA-256.
7. Encrypt the package before moving it into durable local retention. The approved implementation uses authenticated encryption/KDF parameters recorded by `encryption_scheme_version`.
7. Encrypt the package before durable local retention with Qlockify's `openssl enc -aes-256-cbc -salt -pbkdf2 -iter 200000` flow. The manifest records the scheme/version and SHA-256 integrity evidence because CBC is not authenticated encryption.
8. Upload over TLS to the private S3-compatible destination. Upload to a temporary object key, verify remote size/checksum/metadata, then finalize the immutable timestamped key.
9. Mark the backup successful only after remote verification. Export success timestamp/duration/size metrics.
9. Mark the backup successful only after remote verification. Record success timestamp, duration, and size in structured MVP evidence; export the same signals as metrics in the required post-MVP observability phase.
10. Retain the newest three verified encrypted local artifacts and seven verified encrypted remote artifacts. Delete stale artifacts only after the new remote object is verified; never delete the last known-good local or remote backup.
11. Release the lock and ensure temporary plaintext files are removed from the exact temporary directory.
@@ -71,7 +73,7 @@ A cleanup failure is a ticket, not a reason to mark the backup failed if the ver
- archive/manifest/encrypted-object checksums and sizes are non-zero and agree
- retention counts are at least one and no greater than approved limits after cleanup
- backup duration/size are within expected bounds or investigated
- no excluded filenames/classes appear in the archive listing
- no excluded filenames/classes appear in the package-content inspection
- local backup directory and remote bucket are not publicly readable
- decryption key is retrievable from separate escrow by authorized operators without exposing it in the check record
@@ -82,10 +84,10 @@ Run monthly and before any approved destructive migration.
1. Select an explicit backup id, not an unexamined implicit `latest`, and record why it was chosen.
2. Provision an empty non-production destination with compatible PostgreSQL and enough free space. Confirm its environment/hostname/database do not match production.
3. Download the exact encrypted object to a new restrictive temporary directory.
4. Verify remote-object checksum/size, decrypt, validate package and archive checksum, parse the manifest, and inspect `pg_restore --list`.
4. Verify remote-object checksum/size, decrypt, validate the package and plain-SQL checksum/header, and parse the manifest.
5. Treat the dump as trusted code only if it came from the approved bucket/path and passes provenance checks. PostgreSQL warns that restore executes statements controlled by the dump source.
6. Create a new empty database from `template0` or the equivalent approved empty-cluster procedure. Do not reuse a partially populated database.
7. Restore with error-stop semantics and reviewed owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets.
7. Restore with `psql -X --set ON_ERROR_STOP=on` and the reviewed transaction/owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets.
8. Check out/build the selected release record if needed, then run its validation suite:
- expected migration/import manifests
- representative table/session/attempt counts and invariants
@@ -109,7 +111,7 @@ Production restore is destructive/exclusive and always requires explicit human a
- approved incident/change record and named operator
- current-state backup when PostgreSQL is readable and doing so will not worsen the incident
If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `pg_restore`.
If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `psql`.
## Production restore sequence
@@ -131,7 +133,7 @@ Do not automatically restore `.env`/secrets, TLS, Redis, or frontend assets from
| Failure | Action |
|---|---|
| Backup lock already held | Exit without starting a second backup; alert if stale |
| `pg_dump` non-zero or unreadable listing | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold |
| `pg_dump` non-zero, empty, or invalid plain-SQL output | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold |
| Encryption failure | Do not persist/upload plaintext; secure exact temp path and investigate |
| Remote upload/checksum failure | Keep encrypted local artifact and all prior remote copies; bounded retry; alert |
| Retention cleanup failure | Keep extra copies; ticket; never broaden deletion globs/paths |