# PostgreSQL backup and restore runbook

- Status: Draft operational contract; scripts not yet implemented
- RPO: 24 hours
- RTO: two hours
- Last updated: 2026-08-20

This runbook adapts Qlockify's logical dump, encrypted S3-compatible upload, local/remote retention, and restore flow while removing environment files from the backup and adding exact-destination/production confirmation guards.

## Scope and exclusions

Back up authoritative PostgreSQL data and the minimum manifest required to validate it. Do not include:

- Redis/AOF/cache/queue data
- `.env` or secret files
- deployment SSH and object-store credentials
- backup decryption key
- TLS private keys or ACME account data
- raw challenge CSV archives
- demo plaintext credential exports
- frontend font/license files
- Prometheus/Grafana runtime volumes or container logs

Redis loss is recovered through empty Redis plus idempotent analytics reconciliation/recomputation, not restore.

## Backup package contract

Each encrypted artifact corresponds to one manifest with:

```text
format_version
backup_id
created_at_utc
completed_at_utc
source_environment
source_host_id
postgres_server_version
dump_client_version
database_name
database_archive_sha256
database_archive_bytes
schema_migration_version
active_release_id
deployment_sha
backend_sha
frontend_sha
encryption_scheme_version
```

The manifest stores no password, connection URL, secret value, license identifier/text, raw font copy, CSV path/content, or merchant data sample. A checksum of the encrypted remote object is recorded separately from the plaintext archive checksum.

## Scheduled backup procedure

1. Acquire a backup lock that conflicts with restore and destructive migration but need not block ordinary application reads/writes.
2. Resolve and verify the exact production host/database identity. Validate free disk for temporary output plus the encrypted artifact.
3. Confirm PostgreSQL health, client/server major compatibility, active three-hash release id, and migration version.
4. Create a consistent logical dump using PostgreSQL 18 `pg_dump` custom or directory format. Capture stderr and fail on a non-zero exit.
5. Inspect the archive table of contents with `pg_restore --list`; reject an empty/truncated/unreadable archive.
6. Build the non-secret manifest and calculate the plaintext archive SHA-256.
7. Encrypt the package before moving it into durable local retention. The approved implementation uses authenticated encryption/KDF parameters recorded by `encryption_scheme_version`.
8. Upload over TLS to the private S3-compatible destination. Upload to a temporary object key, verify remote size/checksum/metadata, then finalize the immutable timestamped key.
9. Mark the backup successful only after remote verification. Export success timestamp/duration/size metrics.
10. Retain the newest three verified encrypted local artifacts and seven verified encrypted remote artifacts. Delete stale artifacts only after the new remote object is verified; never delete the last known-good local or remote backup.
11. Release the lock and ensure temporary plaintext files are removed from the exact temporary directory.

A cleanup failure is a ticket, not a reason to mark the backup failed if the verified new copy exists. An upload/verification failure is a backup failure and preserves all prior copies.

## Daily verification

- last verified remote backup age is under 30 hours
- archive/manifest/encrypted-object checksums and sizes are non-zero and agree
- retention counts are at least one and no greater than approved limits after cleanup
- backup duration/size are within expected bounds or investigated
- no excluded filenames/classes appear in the archive listing
- local backup directory and remote bucket are not publicly readable
- decryption key is retrievable from separate escrow by authorized operators without exposing it in the check record

## Clean-destination restore rehearsal

Run monthly and before any approved destructive migration.

1. Select an explicit backup id, not an unexamined implicit `latest`, and record why it was chosen.
2. Provision an empty non-production destination with compatible PostgreSQL and enough free space. Confirm its environment/hostname/database do not match production.
3. Download the exact encrypted object to a new restrictive temporary directory.
4. Verify remote-object checksum/size, decrypt, validate package and archive checksum, parse the manifest, and inspect `pg_restore --list`.
5. Treat the dump as trusted code only if it came from the approved bucket/path and passes provenance checks. PostgreSQL warns that restore executes statements controlled by the dump source.
6. Create a new empty database from `template0` or the equivalent approved empty-cluster procedure. Do not reuse a partially populated database.
7. Restore with error-stop semantics and reviewed owner/privilege behavior. Preserve complete stdout/stderr in a restricted evidence path without secrets.
8. Check out/build the selected release record if needed, then run its validation suite:
   - expected migration/import manifests
   - representative table/session/attempt counts and invariants
   - at least two merchant logins and cross-tenant denial
   - payer-card keys absent from API/log output
   - Redis initially empty, analytics recomputation succeeds, four sections/SSE work
   - frontend and API compatibility for the manifest's release
9. Record elapsed time, achieved RPO, achieved RTO, sizes/counts, validation result, operator, and remediation.
10. Destroy rehearsal resources only after evidence is retained and the exact destination is revalidated.

## Production restore approval guard

Production restore is destructive/exclusive and always requires explicit human approval. The future implementation must require all of these independent inputs:

- `RESTORE_ENVIRONMENT=production`
- exact expected host identity matching the current provider instance
- exact expected database name
- explicit backup id/object key (no automatic `latest`)
- a typed confirmation phrase containing both production and the backup id
- active maintenance mode and deployment/restore lock
- approved incident/change record and named operator
- current-state backup when PostgreSQL is readable and doing so will not worsen the incident

If any input is absent or mismatched, exit before stopping writers, dropping/renaming a database, extracting over a target, or running `pg_restore`.

## Production restore sequence

1. Announce maintenance and stop public writes/new sessions. Preserve an operator-only health path.
2. Acquire the exclusive deployment/restore lock; stop worker before backend so no jobs write during restore.
3. Capture the active/previous three-hash release records, database identity, incident timestamps, and current-state backup when safe.
4. Perform all download/decrypt/checksum/archive/provenance checks from the rehearsal procedure before mutating database state.
5. Restore into a newly created empty database/cluster. Do not pipe an unchecked dump directly into the active production database and do not recursively delete a computed volume/path.
6. Validate restore completeness and application invariants using the three recorded repository commits compatible with the backup manifest. Apply only explicitly approved forward migrations.
7. Start with empty Redis. Start backend/worker, confirm readiness, idempotent stale-run reconciliation, cold-cache analytics, sessions, and SSE behavior.
8. Switch application connectivity to the validated restored destination using the approved implementation strategy.
9. Run public TLS/login/dashboard plus private database/queue/tenant smoke checks. Observe for 15 minutes.
10. Reopen traffic, mark the restore complete, and schedule a fresh backup. Retain the old database until the incident owner approves disposal under a separate exact-target procedure.

Do not automatically restore `.env`/secrets, TLS, Redis, or frontend assets from a database backup. Reconstruct platform configuration from Git/release records and secrets from separate escrow.

## Failure handling

| Failure | Action |
|---|---|
| Backup lock already held | Exit without starting a second backup; alert if stale |
| `pg_dump` non-zero or unreadable listing | Delete only the exact failed temporary artifact; keep prior backups; page on age threshold |
| Encryption failure | Do not persist/upload plaintext; secure exact temp path and investigate |
| Remote upload/checksum failure | Keep encrypted local artifact and all prior remote copies; bounded retry; alert |
| Retention cleanup failure | Keep extra copies; ticket; never broaden deletion globs/paths |
| Manifest/archive mismatch | Quarantine artifact and select another known-good backup |
| Restore SQL/object failure | Keep traffic closed, preserve logs/destination, do not rerun blindly into partial DB |
| Post-restore invariant failure | Do not switch/reopen; return to old database if safe or select another backup |
| Missing decryption key | Stop; encrypted backup is unrecoverable without the separately escrowed key |
| Wrong environment/host/database confirmation | Abort before mutation and record denied attempt |

## Acceptance evidence

- [ ] Daily backup completed and verified remotely within 30 hours
- [ ] Backup contains PostgreSQL archive plus non-secret manifest only
- [ ] Three local/seven remote verified encrypted artifacts retained
- [ ] Restore rehearsal uses an explicit backup id and clean non-production database
- [ ] Archive trust/checksum/listing/compatibility checks precede restore
- [ ] Application, tenant, session, cold-cache, worker, and SSE validations pass
- [ ] Achieved RPO <=24 hours and RTO <=2 hours recorded
- [ ] Every missing production confirmation factor aborts before mutation
- [ ] No Redis, env, secrets, TLS keys, CSV, credentials, font/license, logs, or monitoring runtime data enter backup