docs: add bootstrap and recovery runbooks
This commit is contained in:
154
docs/runbooks/BACKUP-RESTORE.md
Normal file
154
docs/runbooks/BACKUP-RESTORE.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# PostgreSQL backup and restore runbook
|
||||
|
||||
- Status: Draft operational contract; scripts not yet implemented
|
||||
- RPO: 24 hours
|
||||
- RTO: two hours
|
||||
- Last updated: 2026-08-20
|
||||
|
||||
This runbook adapts Qlockify's logical dump, encrypted S3-compatible upload, local/remote retention, and restore flow while removing environment files from the backup and adding exact-destination/production confirmation guards.
|
||||
|
||||
## Scope and exclusions
|
||||
|
||||
Back up authoritative PostgreSQL data and the minimum manifest required to validate it. Do not include:
|
||||
|
||||
- Redis/AOF/cache/queue data
|
||||
- `.env` or secret files
|
||||
- deployment SSH/registry/object-store credentials
|
||||
- backup decryption key
|
||||
- TLS private keys or ACME account data
|
||||
- raw challenge CSV archives
|
||||
- demo plaintext credential exports
|
||||
- frontend font/license files
|
||||
- Prometheus/Grafana runtime volumes or container logs
|
||||
|
||||
Redis loss is recovered through empty Redis plus idempotent analytics reconciliation/recomputation, not restore.
|
||||
|
||||
## Backup package contract
|
||||
|
||||
Each encrypted artifact corresponds to one manifest with:
|
||||
|
||||
```text
|
||||
format_version
|
||||
backup_id
|
||||
created_at_utc
|
||||
completed_at_utc
|
||||
source_environment
|
||||
source_host_id
|
||||
postgres_server_version
|
||||
dump_client_version
|
||||
database_name
|
||||
database_archive_sha256
|
||||
database_archive_bytes
|
||||
schema_migration_version
|
||||
active_release_id
|
||||
deployment_sha
|
||||
backend_sha
|
||||
frontend_sha
|
||||
encryption_scheme_version
|
||||
```
|
||||
|
||||
The manifest stores no password, connection URL, secret value, license identifier/text, raw font copy, CSV path/content, or merchant data sample. A checksum of the encrypted remote object is recorded separately from the plaintext archive checksum.
|
||||
|
||||
## Scheduled backup procedure
|
||||
|
||||
1. Acquire a backup lock that conflicts with restore and destructive migration but need not block ordinary application reads/writes.
|
||||
2. Resolve and verify the exact production host/database identity. Validate free disk for temporary output plus the encrypted artifact.
|
||||
3. Confirm PostgreSQL health, client/server major compatibility, active release id, and migration version.
|
||||
4. Create a consistent logical dump using PostgreSQL 18 `pg_dump` custom or directory format. Capture stderr and fail on a non-zero exit.
|
||||
5. Inspect the archive table of contents with `pg_restore --list`; reject an empty/truncated/unreadable archive.
|
||||
6. Build the non-secret manifest and calculate the plaintext archive SHA-256.
|
||||
7. Encrypt the package before moving it into durable local retention. The approved implementation uses authenticated encryption/KDF parameters recorded by `encryption_scheme_version`.
|
||||
8. Upload over TLS to the private S3-compatible destination. Upload to a temporary object key, verify remote size/checksum/metadata, then finalize the immutable timestamped key.
|
||||
9. Mark the backup successful only after remote verification. Export success timestamp/duration/size metrics.
|
||||
10. Retain the newest three verified encrypted local artifacts and seven verified encrypted remote artifacts. Delete stale artifacts only after the new remote object is verified; never delete the last known-good local or remote backup.
|
||||
11. Release the lock and ensure temporary plaintext files are removed from the exact temporary directory.
|
||||
|
||||
A cleanup failure is a ticket, not a reason to mark the backup failed if the verified new copy exists. An upload/verification failure is a backup failure and preserves all prior copies.
|
||||
|
||||
## Daily verification
|
||||
|
||||
- last verified remote backup age is under 30 hours
|
||||
- archive/manifest/encrypted-object checksums and sizes are non-zero and agree
|
||||
- retention counts are at least one and no greater than approved limits after cleanup
|
||||
- backup duration/size are within expected bounds or investigated
|
||||
- no excluded filenames/classes appear in the archive listing
|
||||
- local backup directory and remote bucket are not publicly readable
|
||||
- decryption key is retrievable from separate escrow by authorized operators without exposing it in the check record
|
||||
|
||||
## Clean-destination restore rehearsal
|
||||
|
||||
Run monthly and before any approved destructive migration.
|
||||
|
||||
1. Select an explicit backup id, not an unexamined implicit `latest`, and record why it was chosen.
|
||||
2. Provision an empty non-production destination with compatible PostgreSQL and enough free space. Confirm its environment/hostname/database do not match production.
|
||||
3. Download the exact encrypted object to a new restrictive temporary directory.
|
||||
4. Verify remote-object checksum/size, decrypt, validate package and archive checksum, parse the manifest, and inspect `pg_restore --list`.
|
||||
5. Treat the dump as trusted code only if it came from the approved bucket/path and passes provenance checks. PostgreSQL warns that restore executes statements controlled by the dump source.
|
||||
6. Create a new empty database from `template0` or the equivalent approved empty-cluster procedure. Do not reuse a partially populated database.
|
||||
7. Restore with error-stop semantics and reviewed owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets.
|
||||
8. Run the selected release's validation suite:
|
||||
- expected migration/import manifests
|
||||
- representative table/session/attempt counts and invariants
|
||||
- at least two merchant logins and cross-tenant denial
|
||||
- payer-card keys absent from API/log output
|
||||
- Redis initially empty, analytics recomputation succeeds, four sections/SSE work
|
||||
- frontend and API compatibility for the manifest's release
|
||||
9. Record elapsed time, achieved RPO, achieved RTO, sizes/counts, validation result, operator, and remediation.
|
||||
10. Destroy rehearsal resources only after evidence is retained and the exact destination is revalidated.
|
||||
|
||||
## Production restore approval guard
|
||||
|
||||
Production restore is destructive/exclusive and always requires explicit human approval. The future implementation must require all of these independent inputs:
|
||||
|
||||
- `RESTORE_ENVIRONMENT=production`
|
||||
- exact expected host identity matching the current provider instance
|
||||
- exact expected database name
|
||||
- explicit backup id/object key (no automatic `latest`)
|
||||
- a typed confirmation phrase containing both production and the backup id
|
||||
- active maintenance mode and deployment/restore lock
|
||||
- approved incident/change record and named operator
|
||||
- current-state backup when PostgreSQL is readable and doing so will not worsen the incident
|
||||
|
||||
If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `pg_restore`.
|
||||
|
||||
## Production restore sequence
|
||||
|
||||
1. Announce maintenance and stop public writes/new sessions. Preserve an operator-only health path.
|
||||
2. Acquire the exclusive deployment/restore lock; stop worker before backend so no jobs write during restore.
|
||||
3. Capture the active/previous release manifests, database identity, incident timestamps, and current-state backup when safe.
|
||||
4. Perform all download/decrypt/checksum/archive/provenance checks from the rehearsal procedure before mutating database state.
|
||||
5. Restore into a newly created empty database/cluster. Do not pipe an unchecked dump directly into the active production database and do not recursively delete a computed volume/path.
|
||||
6. Validate restore completeness and application invariants using the release compatible with the backup manifest. Apply only explicitly approved forward migrations.
|
||||
7. Start with empty Redis. Start backend/worker, confirm readiness, idempotent stale-run reconciliation, cold-cache analytics, sessions, and SSE behavior.
|
||||
8. Switch application connectivity to the validated restored destination using the approved implementation strategy.
|
||||
9. Run public TLS/login/dashboard plus private database/queue/tenant smoke checks. Observe for 15 minutes.
|
||||
10. Reopen traffic, mark the restore complete, and schedule a fresh backup. Retain the old database until the incident owner approves disposal under a separate exact-target procedure.
|
||||
|
||||
Do not automatically restore `.env`/secrets, TLS, Redis, or frontend assets from a database backup. Reconstruct platform configuration from Git/release records and secrets from separate escrow.
|
||||
|
||||
## Failure handling
|
||||
|
||||
| Failure | Action |
|
||||
|---|---|
|
||||
| Backup lock already held | Exit without starting a second backup; alert if stale |
|
||||
| `pg_dump` non-zero or unreadable listing | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold |
|
||||
| Encryption failure | Do not persist/upload plaintext; secure exact temp path and investigate |
|
||||
| Remote upload/checksum failure | Keep encrypted local artifact and all prior remote copies; bounded retry; alert |
|
||||
| Retention cleanup failure | Keep extra copies; ticket; never broaden deletion globs/paths |
|
||||
| Manifest/archive mismatch | Quarantine artifact and select another known-good backup |
|
||||
| Restore SQL/object failure | Keep traffic closed, preserve logs/destination, do not rerun blindly into partial DB |
|
||||
| Post-restore invariant failure | Do not switch/reopen; return to old database if safe or select another backup |
|
||||
| Missing decryption key | Stop; encrypted backup is unrecoverable without the separately escrowed key |
|
||||
| Wrong environment/host/database confirmation | Abort before mutation and record denied attempt |
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
- [ ] Daily backup completed and verified remotely within 30 hours
|
||||
- [ ] Backup contains PostgreSQL archive plus non-secret manifest only
|
||||
- [ ] Three local/seven remote verified encrypted artifacts retained
|
||||
- [ ] Restore rehearsal uses an explicit backup id and clean non-production database
|
||||
- [ ] Archive trust/checksum/listing/compatibility checks precede restore
|
||||
- [ ] Application, tenant, session, cold-cache, worker, and SSE validations pass
|
||||
- [ ] Achieved RPO <=24 hours and RTO <=2 hours recorded
|
||||
- [ ] Every missing production confirmation factor aborts before mutation
|
||||
- [ ] No Redis, env, secrets, TLS keys, CSV, credentials, font/license, logs, or monitoring runtime data enter backup
|
||||
Reference in New Issue
Block a user