zarinpal-analytics-deployment/tasks/todo.md

# Platform Operations implementation tasks

- Status: Approved — Phase 4 implementation authorized 2026-08-20
- Canonical plan: `tasks/plan.md`
- Adaptation baseline: `docs/QLOCKIFY-ADAPTATION-CONTRACT.md`

## Task 0: Approve the implementation plan

**Description:** Confirm that the Qlockify adaptation contract, dependency order, task boundaries, and verification gates are the correct basis for later implementation.

**Acceptance criteria:**
- [x] Platform Operations specification and ADRs 0001–0003 are approved.
- [x] Qlockify flow/environment mapping is documented without real runtime values.
- [x] Stakeholder approved this plan/task breakdown and authorized implementation on 2026-08-20.

**Verification:** Documentation diff review and confirmation that no implementation artifact was created.

**Dependencies:** None

**Files likely touched:** documentation only

**Estimated scope:** Small

## Task 1: Add the deployment environment contract

**Description:** Add the root deployment example and redaction-safe validation for the Qlockify-shaped root/backend/frontend environment layout. Backend and frontend examples remain owned by their repositories and are consumed without duplication.

**Acceptance criteria:**
- [ ] The root example contains only deployment-owned safe defaults/blank placeholders; every service-owned variable has one documented owner.
- [ ] Real `.env` files stay ignored, mode `0600`, excluded from backup, and absent from rendered/logged output.
- [ ] Validation rejects missing, unknown, malformed, or client-exposed secret variables for the selected operation.

**Verification:** Placeholder render, negative validation matrix, Git-ignore/permission checks, and staged secret scan.

**Dependencies:** Task 0 plus explicit implementation authorization

**Files likely touched:** `.env.example`, validation helper/test, documentation; maximum 5 files

**Estimated scope:** Medium

## Task 2: Add PostgreSQL and Redis Compose foundation

**Description:** Define pinned private data services, health checks, named volumes, authentication, resource limits, bounded logs, and Redis AOF behavior.

**Acceptance criteria:**
- [ ] PostgreSQL and Redis become healthy using the approved pins and memory limits.
- [ ] Neither publishes a host port; only approved services can join the private data network.
- [ ] Redis uses AOF `everysec`, `noeviction`, 192-MB maxmemory, and remains excluded from backup.

**Verification:** Compose validation, health/port/network/volume/resource inspection, restart and Redis-loss tests.

**Dependencies:** Task 1

**Files likely touched:** `compose.yaml`, data config, focused validation/test files; maximum 5 files

**Estimated scope:** Medium

## Task 3: Add backend runtime boundaries

**Description:** Wire API and worker services plus one-off migrate/import/admin profiles from the same recorded backend checkout/image lineage.

**Acceptance criteria:**
- [ ] API and worker use private networks, health/readiness, graceful shutdown, and approved resource limits.
- [ ] Migrate/import/admin are explicit one-off profiles and never run during ordinary startup.
- [ ] Worker uses pre-1.0 Asynq `v0.26.0` with concurrency 16 globally and 4 per merchant.

**Verification:** Compose profile/startup/failure/shutdown tests and service command-contract review.

**Dependencies:** Task 2 and the backend-owned command contract

**Files likely touched:** `compose.yaml`, backend image definition, lifecycle tests/docs; maximum 5 files

**Estimated scope:** Medium

## Task 4: Add frontend production build

**Description:** Build and serve the direct frontend checkout using its ordinary static context and same-origin API configuration.

**Acceptance criteria:**
- [ ] Build consumes frontend commit `64f5ac2` or later and its tracked font/license normally.
- [ ] Static sentinel, SPA fallback, and emitted font resource/content type pass.
- [ ] Deployment Git contains no font/license copy, fetch, credentials, or alternate asset path.

**Verification:** Production build/static-server checks plus repository-policy scan.

**Dependencies:** Task 1 and the frontend-owned build contract

**Files likely touched:** frontend image definition, `compose.yaml`, focused health/tests; maximum 5 files

**Estimated scope:** Medium

## Checkpoint A: Runtime foundation

- [ ] Tasks 1–4 pass from clean nested checkouts with placeholder test values.
- [ ] Normal startup cannot invoke migration/import/admin.
- [ ] No secrets, runtime state, raw data, or duplicated font assets are tracked.

## Task 5: Adapt Qlockify Nginx routing

**Description:** Adapt the Qlockify template/entrypoint for the single canonical domain, default-host rejection, frontend/API routing, security headers, health, caching, and SSE.

**Acceptance criteria:**
- [ ] Only ports 80/443 publish; HTTP redirects exactly to `zarinpal.amiirkhl.ir`; unknown hosts reject.
- [ ] API and frontend routing/fallback do not overlap and public readiness remains generic.
- [ ] Exact SSE route preserves auth/event ids and streams heartbeats without buffering, cache, compression, or premature timeout.

**Verification:** `nginx -t`, route/header/default-host tests, static/font test, and streamed SSE fixture.

**Dependencies:** Tasks 3–4

**Files likely touched:** Nginx template, image/entrypoint, focused tests; maximum 5 files

**Estimated scope:** Medium

## Task 6: Adapt Qlockify Certbot lifecycle

**Description:** Adapt Qlockify's HTTP-01 bootstrap, pinned Certbot service, certificate watcher, renewal, and Nginx reload for the single domain.

**Acceptance criteria:**
- [ ] Missing production certificate uses only the bounded bootstrap certificate path until issuance.
- [ ] Staging issuance, production issuance, renewal, and validated Nginx reload pass.
- [ ] No alternate domain, local production certificate mode, private key, or ACME state enters Git.

**Verification:** Compose/entrypoint tests, staging ACME drill, certificate name/expiry/protocol inspection.

**Dependencies:** Task 5

**Files likely touched:** Certbot entrypoint/config, `compose.yaml`, focused tests/docs; maximum 5 files

**Estimated scope:** Medium

## Task 7: Add guarded first-start orchestration

**Description:** Add the explicit first-server path that starts data services, migrates, imports/reconciles the pinned CSV, and activates public services only after success.

**Acceptance criteria:**
- [ ] Input is exact/read-only and migration/import/reconciliation failures prevent public activation.
- [ ] A completed identical import is a safe no-op; recovery never guesses a partial CSV position.
- [ ] Ordinary deploy/start/reboot paths never invoke first-start, importer, or admin jobs.

**Verification:** Success, checksum/schema/count, interruption/recovery, idempotency, and ordinary-start negative tests.

**Dependencies:** Tasks 2–6 and approved backend importer commands

**Files likely touched:** `scripts/first-start.sh`, Compose profile, focused tests/docs; maximum 5 files

**Estimated scope:** Medium

## Checkpoint B: Edge and first-start

- [ ] Nginx/TLS/static/API/SSE checks pass.
- [ ] First-start is fail-closed and ordinary startup never imports.
- [ ] Only Nginx is externally reachable.

## Task 8: Add release records and operation lock

**Description:** Define ignored current/previous/timestamped release records and one exclusive lock shared by deploy, rollback, restore, migration, and import as required.

**Acceptance criteria:**
- [ ] Record captures three SHAs, local image ids, migration, backup, operator, health, outcome, and previous release without secrets.
- [ ] Invalid/incomplete records cannot become active and current/previous updates are atomic.
- [ ] Conflicting operations fail safely without stale-lock ambiguity.

**Verification:** Schema/parser tests, invalid-record matrix, lock contention/stale-owner tests, and redaction scan.

**Dependencies:** Task 1

**Files likely touched:** release schema/helper/tests/docs; maximum 5 files

**Estimated scope:** Medium

## Task 9: Adapt Qlockify manual deployment

**Description:** Adapt Qlockify's deploy structure to verify three manually pulled repositories, build/migrate/replace services, observe health, and activate one coherent release.

**Acceptance criteria:**
- [ ] Script performs no Git fetch/pull/reset and rejects wrong path/remote/branch/SHA/dirty state.
- [ ] Build, migration, readiness, font, worker, or SSE failure leaves the active release unchanged and records evidence.
- [ ] Successful deployment records all three SHAs and passes a 15-minute postflight observation before activation.

**Verification:** ShellCheck, disposable Git fixtures, Compose mocks/integration, and induced failure matrix.

**Dependencies:** Tasks 5–8

**Files likely touched:** `scripts/deploy.sh`, release helper, focused tests/runbook; maximum 5 files

**Estimated scope:** Medium

## Task 10: Add three-hash rollback

**Description:** Select the previous release record, verify its three revisions, rebuild, and run the normal postflight without automatic database down-migration.

**Acceptance criteria:**
- [ ] Exact previous hashes and clean repository guards precede checkout/build.
- [ ] Migration incompatibility stops rollback with the current stack preserved.
- [ ] Normal application rollback completes within 15 minutes and is recorded as a new event.

**Verification:** Git fixture tests, incompatible-migration negative test, and timed rollback drill.

**Dependencies:** Task 9

**Files likely touched:** rollback script/helper/tests/runbook; maximum 5 files

**Estimated scope:** Medium

## Checkpoint C: Release safety

- [ ] Dirty/diverged/manual-pull/build/migration/readiness failures preserve the active stack.
- [ ] Active and previous releases map to exact three-repository revisions.
- [ ] Timed rollback meets its target without a down migration.

## Task 11: Adapt Qlockify PostgreSQL backup

**Description:** Retain Qlockify's matching-client plain-SQL dump flow while packaging only PostgreSQL plus a non-secret release/migration/checksum manifest.

**Acceptance criteria:**
- [ ] Dump nonzero exit, empty output, invalid header, or checksum failure prevents package success.
- [ ] Package contains only plain SQL and the approved non-secret manifest/checksums.
- [ ] `.env`, secrets, Redis, TLS, CSV, credentials, logs, monitoring, font/license, and media are absent.

**Verification:** Fixture database dump, package listing/header/checksum checks, and forbidden-content scan.

**Dependencies:** Tasks 2, 8

**Files likely touched:** `scripts/backup.sh`, manifest helper/tests/runbook; maximum 5 files

**Estimated scope:** Medium

## Task 12: Adapt encrypted ParsPack upload and retention

**Description:** Retain Qlockify's OpenSSL AES-256-CBC PBKDF2-200k and rclone structure while keeping three encrypted local and seven encrypted remote verified backups.

**Acceptance criteria:**
- [ ] Upload uses a private temporary key and is finalized only after remote verification.
- [ ] Failed encryption/upload/verification preserves all previous copies and no plaintext backup remains durably.
- [ ] Retention removes only exact stale targets after new-copy verification and never deletes the last good copy.

**Verification:** Fake S3/ParsPack-compatible fixture, failure/retention matrix, checksum and plaintext-residue checks.

**Dependencies:** Task 11

**Files likely touched:** upload script, retention helper/tests/runbook; maximum 5 files

**Estimated scope:** Medium

## Task 13: Add guarded local restore

**Description:** Restore an explicitly selected verified backup into a new empty target using `psql -X --set ON_ERROR_STOP=on` and independent production guards.

**Acceptance criteria:**
- [ ] Provenance/checksum/header/version/free-space/exact-empty-target checks precede mutation.
- [ ] Every missing production factor aborts before writers stop or a database is changed.
- [ ] Restore never changes `.env`, Redis, TLS, CSV, credentials, logs, or frontend assets.

**Verification:** Disposable database restore, negative guard matrix, partial-SQL failure, and exclusion checks.

**Dependencies:** Task 11

**Files likely touched:** `scripts/restore.sh`, guard/helper/tests/runbook; maximum 5 files

**Estimated scope:** Medium

## Task 14: Adapt encrypted S3 restore handoff

**Description:** Download an explicit encrypted object, verify/decrypt it in a restrictive temporary directory, and hand it to the guarded local restore.

**Acceptance criteria:**
- [ ] Production requires an explicit object id; an implicit `latest` cannot select a production backup.
- [ ] Remote identity/size/checksum and decrypted package checks pass before restore invocation.
- [ ] Temporary encrypted/plaintext material is removed only from the exact validated temporary directory.

**Verification:** Fake S3 object-selection/tamper tests, decryption failure tests, and guarded handoff rehearsal.

**Dependencies:** Tasks 12–13

**Files likely touched:** `scripts/restore-from-s3.sh`, focused tests/runbook; maximum 4 files

**Estimated scope:** Medium

## Checkpoint D: Recovery

- [ ] Daily schedule/3-local/7-remote retention behaves safely.
- [ ] Forbidden content is absent from every backup package.
- [ ] Clean restore passes schema/count/tenant/login/cold-cache/worker/SSE validation within two hours.

## Task 15: Bootstrap and accept the MVP server

**Description:** Execute the reviewed runbooks on the prepared Ubuntu server, fill runtime values, perform first-start, activate the release, and rehearse rollback/restore.

**Acceptance criteria:**
- [ ] Host access/firewall/swap/Docker/DNS/TLS and only-80/443 exposure checks pass.
- [ ] Migration/import/accounts/tenant isolation/worker/SSE/frontend font and manual health/log checks pass.
- [ ] Three-hash release, rollback, encrypted remote backup, and clean restore meet their time/recovery targets.

**Verification:** Signed non-secret bootstrap/MVP acceptance record.

**Dependencies:** Checkpoints A–D and a prepared server/runtime values

**Files likely touched:** ignored server evidence/release records only

**Estimated scope:** Operational session

## Checkpoint E: MVP ready

- [ ] Every MVP acceptance criterion passes.
- [ ] Active/previous releases and backup/restore are proven.
- [ ] Stakeholder explicitly opens the observability phase.

## Task 16: Add private Prometheus collection

**Description:** Add post-MVP application, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS collection with bounded labels and retention.

**Acceptance criteria:**
- [ ] Required signals and target health pass within approved resource/retention limits.
- [ ] Prometheus/exporters are private and not routed by Nginx.
- [ ] Merchant/user/request/card identifiers are absent from metric labels.

**Verification:** Target/signal/cardinality/resource tests and sensitive-label scan.

**Dependencies:** Checkpoint E

**Files likely touched:** split into focused Prometheus/exporter slices of no more than 5 files

**Estimated scope:** Multiple Medium slices

## Task 17: Add private Grafana dashboards and alerts

**Description:** Add post-MVP dashboards, release correlation, actionable symptom alerts, runbook links, and tested operator delivery.

**Acceptance criteria:**
- [ ] Service, worker/SSE, data, host, backup, and TLS dashboards answer the approved operator questions.
- [ ] Every page/ticket alert is actionable, bounded, linked to a runbook, and tested through an induced failure.
- [ ] Grafana remains private and contains no sensitive labels, logs, or embedded credentials.

**Verification:** Dashboard review, induced failure, alert test-fire, access and redaction checks.

**Dependencies:** Task 16

**Files likely touched:** split into focused dashboard/alert slices of no more than 5 files

**Estimated scope:** Multiple Medium slices

## Final observability checkpoint before AI

- [ ] Prometheus/Grafana acceptance criteria pass after MVP.
- [ ] Alert delivery and runbook ownership are proven.
- [ ] Stakeholder explicitly approves beginning AI implementation.
- [ ] No CI/CD pipeline or automated deployment system was introduced.