diff --git a/tasks/plan.md b/tasks/plan.md index 851ecb6..b3385de 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -1,139 +1,125 @@ # Implementation Plan: Platform Operations -- Status: Planned from draft specification; implementation blocked pending explicit approval +- Status: Planned from draft revision; implementation blocked pending explicit spec approval - Capability: Platform operations - Task list target: `tasks/todo.md` - Last updated: 2026-08-20 -## Overview +## Confirmed implementation direction -Implement the approved Platform Operations contract as a single-server, production-shaped Docker Compose platform for `zarinpal.amiirkhl.ir`. The deployment repository remains the only operational coordinator; backend and frontend source stay in their own repositories and publish immutable images. Work proceeds from decisions and validation contracts through topology, edge, lifecycle, release automation, recovery, and end-to-end proof. +Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds. -No task below is authorized merely because it appears in this plan. The Platform Operations spec must be approved and its blocking decisions resolved first. +There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work. + +No task below is authorized merely because it appears here. ## Dependency graph ```text -approve spec + resolve platform versions/host/runner/backup decisions +approve revised spec + resolve server/runtime contracts | - +--> release manifest + configuration/secret schemas + +--> safe config + manual release-record contract | | - | +--> Compose networks/data services + | +--> Compose data/app services | | | - | | +--> migration/import/admin lifecycle - | | +--> API/worker/frontend services + | | +--> migrate/import/admin separation + | | +--> manual deploy/rollback | | - | +--> image/build contracts in service repositories - | | - | +--> cross-repository candidate events + | +--> Nginx/Certbot routing and TLS | - +--> Nginx/TLS/routing contract implementation + +--> Qlockify backup adaptation --> guarded restore --> rehearsal | - +--> observability baseline - | - +--> backup --> guarded restore --> restore rehearsal - | - +--> host bootstrap validation - | - v - full release/rollback/DR acceptance + +--> first-server bootstrap --> MVP acceptance + | + v + Prometheus/Grafana phase + | + v + AI may begin ``` ## Architecture decisions -- Preserve ADR 0001's three-repository topology. -- Build application images once in service CI and deploy immutable digests through ADR 0002 release manifests. -- Consume the licensed font/license through the ordinary frontend context per accepted ADR 0003 and frontend commit `64f5ac2`; no deployment injection path. -- Keep PostgreSQL authoritative and backed up. Redis uses routine AOF persistence but remains excluded from disaster-recovery backup. -- Separate API, worker, migration, one-time import, and administrative commands. No recurring CSV or scheduled application worker exists. -- Use expand/contract migrations and application-image rollback; never automatic database down migrations. -- Keep Nginx as the only public entry point and monitoring/operator access private. +- Preserve ADR 0001's three independent repositories and ignored nested service checkouts. +- Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records. +- Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license. +- Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags. +- Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup. +- Separate API, worker, migration, one-time CSV import, and administrative processes. +- Keep Asynq pinned to pre-1.0 `v0.26.0`. +- Use automatic Let's Encrypt/Certbot TLS. +- Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI. -## Phases and checkpoints +## Phase 0: approve exact contracts -### Phase 0: approval and evidence +- Approve the revised Platform Operations spec. +- Resolve server/provider/capacity/user/path, image tags, resource limits, repository URLs/commands/health routes, backup endpoint/credentials/schedule, Redis AOF choice, release-record format, operator/secret escrow, and MVP-ready/observability gates. +- Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery. +- Obtain the frontend build/static health contract. -- Resolve host/provider/capacity, container privilege model, forge/runner/registry versions, exact image versions/digests, resource budgets, API health/routes, backup encryption/provider, certificate-renewal method, and operator ownership. -- Approve `docs/PLATFORM-OPERATIONS-SPEC.md` and ADR 0002. ADR 0003 is already accepted. -- Obtain approved machine-readable backend/frontend interface and runtime command contracts. +Checkpoint: no implementation file exists before explicit approval and resolved inputs. -Checkpoint: no implementation file exists before the approval record and decisions are complete. +## Phase 1: configuration and Compose foundation -### Phase 1: release and configuration foundation +- Add safe tracked configuration examples and validate protected external secret paths. +- Define non-secret active/previous release records containing the three SHAs and health/migration evidence. +- Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles. +- Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts. -- Define and validate the release-manifest schema, active/previous release records, exact three-repository/image provenance, and compatibility fields. -- Define tracked safe configuration examples plus the external host secret-file contract. -- Add policy tests that reject floating images, unintended ports, secret-like values, missing health checks, or font injection paths. +Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin. -Checkpoint: a synthetic manifest/config validates offline and contains no secret values or mutable references. +## Phase 2: Nginx, TLS, and lifecycle -### Phase 2: runtime topology and edge +- Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`. +- Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior. +- Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset. +- Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release. -- Implement isolated Compose networks, named volumes, PostgreSQL/Redis health/resources, private application services, and monitoring profile. -- Implement one-off migration, import, and administrative profiles without coupling them to normal startup. -- Implement canonical-domain Nginx/TLS, SPA/API separation, exact SSE route behavior, security headers, limits, and health routing. -- Add focused topology/routing tests before deployment automation. +Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes. -Checkpoint: local/integration inspection proves only Nginx publishes 80/443, startup gates are correct, and SSE is not buffered. +## Phase 3: backup, restore, and first-server acceptance -### Phase 3: service artifact and lifecycle coordination +- Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials. +- Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only. +- Add exact destination and production confirmation guards. +- Bootstrap the server, deploy the MVP, perform one-time import/credential setup, create an encrypted remote backup, and rehearse clean restore within two hours. -- Coordinate backend image commands/health/labels and frontend static/font health with the owning repositories through approved changes there; do not duplicate their source here. -- Implement deployment locks, preflight, migration, service replacement, postflight, observation, and application rollback using release manifests. -- Implement Redis-loss reconciliation acceptance with backend/worker owners, preserving Asynq `v0.26.0` as a pinned pre-1.0 dependency. +Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks. -Checkpoint: an induced migration/readiness/font/worker failure preserves or restores the prior release. +## Phase 4: required observability before AI -### Phase 4: CI/CD and provenance +- Add Prometheus, Grafana, and minimum private exporters/instrumentation. +- Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards. +- Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery. +- Validate the observability acceptance gate before any AI implementation starts. -- Add deployment-repository validation and production-release workflows. -- Coordinate service-repository workflows that publish immutable images and send candidate events only after required checks. -- Enforce protected environment approval, serialized release, registry digest retention, SBOM/provenance/vulnerability gates, and secret isolation. - -Checkpoint: PRs never deploy; backend/frontend candidate events cannot directly mutate production; one manifest controls a tested release and rollback. - -### Phase 5: observability and recovery - -- Implement bounded structured container logging, Prometheus/exporters/Grafana, dashboards, and symptom alerts with runbook links. -- Implement encrypted logical PostgreSQL backup, remote verification, three-local/seven-remote retention, and telemetry. -- Implement exact-destination guarded restore into an empty destination and application validation. -- Rehearse restore and record achieved RPO/RTO. - -Checkpoint: induced failures are diagnosable from telemetry, backup age alert fires, and clean restore succeeds within two hours. - -### Phase 6: bootstrap and full acceptance - -- Convert the reviewed first-server checklist into only the minimum approved automation, retaining manual safety/identity gates. -- Bootstrap the selected host, stage DNS/TLS, deploy a release, run one-time import/credentials, and complete alert/backup/restore handoff. -- Execute every measurable criterion from the Platform Operations spec and retain non-sensitive evidence. - -Checkpoint: stakeholder review confirms release, rollback, Redis-loss, backup/restore, exposure, SLO, and operator-boundary evidence. +Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator. ## Risks and mitigations | Risk | Impact | Mitigation | |---|---|---| -| Single-host failure | Full outage | Immutable rebuild path, encrypted off-host PostgreSQL backup, two-hour restore rehearsal, documented DNS/host replacement | -| Cross-repository race | Incompatible release | Candidate events only, serialized deployment lock, exact manifest, compatibility metadata, one coordinator | -| Migration blocks rollback | Extended outage/data risk | Expand/contract, measured locks, fresh verified backup, no automatic down migration | -| Redis loss drops in-flight queue/cache | Partial analytics interruption | AOF for routine restart, PostgreSQL authority, idempotent stale-run reconciliation, cold-cache tests | -| SSE proxy buffering/timeouts | Dashboard appears stuck | Exact route, buffering/compression/cache off, heartbeat-aware timeout, integration test | -| Docker privilege compromise | Host root compromise | Approve rootless/dedicated model, restricted machine identity/SSH, no daemon TCP exposure, minimal host users | -| Secrets leak through Compose/log/backup | Credential/data compromise | File-mounted least privilege, redaction tests, no env in backup, rotate-first incident rule | -| Backup exists but is unusable | Data loss beyond RTO | Archive/checksum/remote verification plus monthly clean restore | -| Font missing/divergent | Broken approved typography/license trail | Frontend-owned tracked inputs at `64f5ac2+`, build/static smoke, digest provenance, no second path | -| Monitoring overload/noise | Lost diagnostics or ignored alerts | Bounded labels/retention, symptom alerts, test-fire, only page/ticket severities | +| Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record | +| Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` | +| Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time | +| Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration | +| Redis loss drops queue/cache | Partial analytics interruption | Optional AOF, PostgreSQL authority, idempotent reconciliation/cold-cache tests | +| SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test | +| Docker-group compromise | Host-root compromise | Dedicated key-only trusted account, restricted SSH, no daemon TCP exposure | +| Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests | +| Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal | +| Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI | -## Parallelization after contracts are fixed +## Parallelization after approval -- Safe in parallel: Nginx focused tests, backup/restore tooling, observability dashboards, and CI validation, provided the shared manifest/config schemas are merged first. -- Sequential: schema migrations, production deploy/rollback, database restore, DNS/TLS cutover, and one-time import. -- Coordinated across repositories: backend commands/health/migration metadata, frontend image/font health, and candidate-event payload. Providers define contracts before consumers implement them. +- Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed. +- Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate. +- Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced. -## Completion gate +## Completion -- Every task in `tasks/todo.md` meets acceptance and verification evidence. -- All spec acceptance criteria pass on a production-shaped host. -- No unresolved decision remains hidden in a default or script. -- Active/previous release manifests, SLO dashboards, alerts, backup, restore rehearsal, and operator handoff are complete. -- Backend/frontend repositories contain only their approved coordinated changes; the deployment repository contains no application source, raw data, secrets, credentials, TLS keys, backups, database volumes, or duplicate font/license files. +- Every authorized task in `tasks/todo.md` passes its acceptance and verification. +- The active stack maps to exact deployment/backend/frontend commits. +- Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass. +- Prometheus/Grafana acceptance passes before AI begins. +- Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets. diff --git a/tasks/todo.md b/tasks/todo.md index 635279d..32119dd 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -1,243 +1,228 @@ # Platform Operations implementation tasks -- Status: Not authorized — wait for explicit approval of the draft spec +- Status: Not authorized — wait for explicit approval of the revised spec - Canonical plan: `tasks/plan.md` -## Task 0: Approve specification and resolve blocking decisions +## Task 0: Approve revision and resolve implementation inputs -**Description:** Review the platform spec/ADRs and record every host, runner, version, route, capacity, backup, TLS, and ownership decision before implementation. +**Description:** Approve the confirmed manual Qlockify-style intent and resolve every remaining server/runtime/backup/phase-gate decision. **Acceptance criteria:** -- [ ] `PLATFORM-OPERATIONS-SPEC.md` status is explicitly approved by the stakeholder. -- [ ] ADR 0002 is accepted or replaced; ADR 0003 remains consistent with frontend commit `64f5ac2`. -- [ ] All remaining platform decisions have named answers/evidence; the frontend font ownership/delivery decision is already resolved by `64f5ac2` and ADR 0003. +- [ ] Revised Platform Operations spec is explicitly approved. +- [ ] ADRs 0001–0003 match the approved topology, manual release, and font decisions. +- [ ] Every remaining decision has a named answer/evidence. -**Verification:** Manual stakeholder approval and clean documentation diff. +**Verification:** Stakeholder approval plus clean documentation review. **Dependencies:** None -**Files likely touched:** `docs/PLATFORM-OPERATIONS-SPEC.md`, `docs/decisions/0002-immutable-cross-repository-releases.md` +**Files likely touched:** spec and ADRs only -**Estimated scope:** Small (1–2 files) +**Estimated scope:** Small -## Task 1: Define release manifest and safe configuration schemas +## Task 1: Define safe configuration and release records -**Description:** Create machine-validated schemas/examples for exact release provenance, compatibility metadata, active/previous pointers, safe public configuration, and external secret files. +**Description:** Define tracked placeholders, external protected secrets, deployment locking, and non-secret active/previous release records. **Acceptance criteria:** -- [ ] Synthetic valid/invalid manifests prove immutable three-repository/image constraints. -- [ ] Examples contain no secret-like values and name every required configuration input. -- [ ] Font provenance is only frontend SHA/image digest; no injection field/path exists. +- [ ] Release record contains three SHAs, local image ids, migration, backup, operator, and health evidence. +- [ ] Configuration examples contain no real or plausible secret values. +- [ ] Font provenance uses frontend commit/build only; no second asset path exists. -**Verification:** Schema tests plus secret scan over staged files. +**Verification:** Schema/format tests, secret scan, invalid-record tests. **Dependencies:** Task 0 -**Files likely touched:** release/config schemas, safe examples, focused tests (maximum 5 files) +**Files likely touched:** examples/schema/tests/docs, maximum 5 files -**Estimated scope:** Medium (3–5 files) +**Estimated scope:** Medium -## Task 2: Implement data-service Compose foundation +## Task 2: Implement PostgreSQL and Redis Compose foundation -**Description:** Add PostgreSQL/Redis pinned services, isolated data network, volumes, authentication, health checks, AOF policy, resources, and no host port publishing. +**Description:** Add pinned PostgreSQL 18 and Redis 7 Alpine, private data network, volumes, authentication, health, resources, and approved Redis AOF choice. **Acceptance criteria:** -- [ ] Compose renders with pinned digests and healthy PostgreSQL/Redis. -- [ ] Redis loss is explicitly non-authoritative and its volume is excluded from backup. -- [ ] External/local port inspection finds no PostgreSQL/Redis host listener. +- [ ] Both services become healthy with explicit limits/tags. +- [ ] Neither publishes a host port. +- [ ] Redis is excluded from backup and loss does not affect PostgreSQL authority. -**Verification:** Compose validation, health inspection, network/port policy tests. +**Verification:** Compose validation, health, port/network, volume, Redis-loss tests. **Dependencies:** Task 1 -**Files likely touched:** Compose files and focused topology tests (maximum 5 files) +**Files likely touched:** Compose/config/tests, maximum 5 files -**Estimated scope:** Medium (3–5 files) +**Estimated scope:** Medium -## Task 3: Implement API, worker, and one-off lifecycle boundaries +## Task 3: Implement backend, worker, and one-off boundaries -**Description:** Wire immutable backend API/worker plus migration/import/admin one-offs with health gates, profiles, locks, graceful stop, and validated configuration. +**Description:** Wire API/worker plus separate migration/import/admin commands and graceful lifecycle. **Acceptance criteria:** -- [ ] Normal startup waits for migration but never runs import/admin automatically. -- [ ] Worker is private/event-driven and retains pinned pre-1.0 Asynq `v0.26.0` contract. -- [ ] Failed migration blocks candidate API/worker; import path is exact/read-only/idempotent. +- [ ] Migration completes before API/worker readiness. +- [ ] Normal startup never runs import/admin; import uses one exact read-only path. +- [ ] Worker is private/event-driven and uses pinned pre-1.0 Asynq `v0.26.0`. -**Verification:** Lifecycle integration tests including failed migration and repeated import fixture. +**Verification:** Startup/failure/import-idempotency/shutdown integration tests. -**Dependencies:** Tasks 1–2 and approved backend command/health contract +**Dependencies:** Task 2 and approved backend commands -**Files likely touched:** Compose lifecycle definitions, wrapper, focused tests (maximum 5 files) +**Files likely touched:** Compose/service wrappers/tests, maximum 5 files per slice -**Estimated scope:** Medium (3–5 files) +**Estimated scope:** Medium -## Task 4: Implement frontend and font artifact health +## Task 4: Implement frontend build and font health -**Description:** Consume the ordinary frontend image/build context and verify static SPA plus tracked font output without copying/injecting assets in deployment. +**Description:** Build the nested frontend checkout normally and verify static SPA/font output without deployment copies or injection. **Acceptance criteria:** -- [ ] Candidate maps to frontend SHA/digest and serves expected font URL/MIME. -- [ ] Missing font/license input fails frontend CI or candidate smoke. -- [ ] Deployment files contain no font binary, license copy, private fetch, or font secret. +- [ ] Build uses recorded frontend commit at/after `64f5ac2`. +- [ ] Expected font resource returns 200 with correct content type. +- [ ] Deployment repository contains no font binary, `FontLicense.txt`, fetch, credential, or alternate path. -**Verification:** Image/static health test and repository policy scan. +**Verification:** Build/static health plus repository-policy scan. -**Dependencies:** Task 1 and approved frontend image contract at/after `64f5ac2` +**Dependencies:** Task 1 and approved frontend build contract -**Files likely touched:** frontend service definition and focused smoke/policy tests (maximum 5 files) +**Files likely touched:** frontend deployment wrapper/service/tests, maximum 5 files -**Estimated scope:** Small/Medium (2–4 files) +**Estimated scope:** Medium -## Task 5: Implement Nginx canonical TLS and routing +## Task 5: Implement Nginx and automatic TLS -**Description:** Add canonical host/default rejection, ACME/TLS, API/frontend separation, exact SSE proxy behavior, security headers, limits, and generic health routes. +**Description:** Adapt Qlockify's Nginx/Certbot pattern for canonical routing, Let's Encrypt, API/frontend separation, exact SSE behavior, security headers, and health. **Acceptance criteria:** -- [ ] Only 80/443 are public; redirects/default-host behavior is exact. -- [ ] SSE buffering/cache/compression are off and heartbeat/replay survives reconnect. -- [ ] API never falls through to SPA; security/TLS/static-cache tests pass. +- [ ] Only 80/443 are public; unknown hosts reject and canonical redirect is exact. +- [ ] Staging/production issuance and renewal/reload pass. +- [ ] SSE heartbeat/reconnect streams without buffering/cache/compression. -**Verification:** `nginx -t`, TLS/header/routing tests, streamed timing/reconnect test. +**Verification:** `nginx -t`, external TLS/header/routing checks, streamed SSE test. -**Dependencies:** Tasks 3–4 and approved API/SSE paths/CSP +**Dependencies:** Tasks 3–4 and approved routes/CSP -**Files likely touched:** Nginx template/entrypoint and focused tests (maximum 5 files) +**Files likely touched:** Nginx/Certbot assets/tests, maximum 5 files per slice -**Estimated scope:** Medium (3–5 files) +**Estimated scope:** Medium -## Checkpoint A: Runtime foundation +## Checkpoint A: Runtime MVP foundation -- [ ] Tasks 1–5 pass together from a clean Compose project. -- [ ] Only Nginx is public and all health/startup boundaries behave as specified. -- [ ] No secret, raw data, backup, TLS key, or duplicate font/license asset is tracked. +- [ ] Tasks 1–5 pass from clean nested checkouts. +- [ ] Only Nginx is public; migration/import/admin/font/SSE boundaries behave correctly. +- [ ] No secret/raw data/runtime/font duplication enters deployment Git. -## Task 6: Implement release preflight, deploy, and rollback +## Task 6: Adapt Qlockify manual deploy and rollback -**Description:** Build the serialized manifest-driven operator workflow for preflight, migration, service replacement, postflight, observation, active/previous switch, and rollback. +**Description:** Modify Qlockify's deploy flow for ZarinPal paths/services and three-hash release records while removing unconditional destructive Git synchronization. **Acceptance criteria:** -- [ ] Deploy uses image digests only and records evidence without secrets. -- [ ] Candidate health/migration/font failure preserves/restores prior release. -- [ ] Application-only rollback completes within 15 minutes and never auto-runs down migration. +- [ ] Latest `origin/main` is fast-forwarded and three hashes recorded before build. +- [ ] Dirty/diverged checkout, pull, build, migration, or health failure preserves the active release. +- [ ] Previous hashes rebuild/restore within 15 minutes without automatic down migration. -**Verification:** Shell/unit tests plus induced staging failures and timed rollback drill. +**Verification:** Script tests plus induced staging failures and timed rollback drill. **Dependencies:** Checkpoint A -**Files likely touched:** deploy/release scripts and focused tests (maximum 5 files per incremental slice) +**Files likely touched:** deploy script/release helper/tests/docs, maximum 5 files per slice -**Estimated scope:** Split into multiple Medium tasks during implementation +**Estimated scope:** Medium -## Task 7: Implement deployment-repository CI/CD +## Task 7: Adapt Qlockify encrypted backup/upload -**Description:** Validate docs/config/Compose/Nginx/scripts/policies, accept authenticated candidates, enforce protected approval/locking, and invoke the manifest-driven release. +**Description:** Reuse Qlockify's PostgreSQL dump, OpenSSL, rclone upload, and retention flow with ZarinPal paths/services/credentials and a non-secret manifest. **Acceptance criteria:** -- [ ] PR workflows validate only and cannot access production secrets/deploy. -- [ ] Production release requires gates plus human approval and is serialized. -- [ ] Provenance/SBOM/scan/secret results attach to the release record. +- [ ] Archive/listing/checksums/encryption/remote verification gate success. +- [ ] Exactly three verified encrypted local and seven remote backups are retained. +- [ ] `.env`, secrets, Redis, TLS, CSV, credentials, logs, monitoring, font, and license are excluded. -**Verification:** Forge event tests with denied PR deploy, concurrent candidate events, failed gate, and approved release. +**Verification:** Fixture database/object-store tests, exclusion inspection, failed-upload/cleanup tests. -**Dependencies:** Task 6 and confirmed forge/runner/registry versions +**Dependencies:** Tasks 1–3 and approved backup inputs -**Files likely touched:** workflow plus CI-focused helpers/tests (maximum 5 files) +**Files likely touched:** adapted backup/upload scripts/tests/docs, maximum 5 files per slice -**Estimated scope:** Medium (3–5 files) +**Estimated scope:** Medium -## Task 8: Coordinate backend/frontend candidate workflows +## Task 8: Adapt Qlockify guarded restore -**Description:** In separately approved service-repository changes, publish one immutable tested image and send candidate metadata without direct production SSH. +**Description:** Reuse Qlockify download/decrypt/restore flow while requiring explicit backup id, exact destination, empty target, and all production guards. **Acceptance criteria:** -- [ ] Backend gates include Go/migration/import/Asynq queue/recovery verification. -- [ ] Frontend gates include Svelte/accessibility/RTL/build/font verification. -- [ ] Service events cannot mutate production and use least-privilege credentials. +- [ ] Every absent/mismatched production factor aborts before mutation. +- [ ] No environment/secret/Redis restoration occurs. +- [ ] Clean restore passes schema/count/tenant/login/cold-cache/worker/SSE checks within two hours. -**Verification:** Each repository's CI on PR/main plus deployment candidate receipt test. +**Verification:** Disposable-database tests, guard negative matrix, timed rehearsal. -**Dependencies:** Task 7 and separate authorization in each service repository +**Dependencies:** Task 7 -**Files likely touched:** owned service workflow/tests only; not this deployment task +**Files likely touched:** adapted restore scripts/tests/docs, maximum 5 files per slice -**Estimated scope:** Separate cross-repository tasks +**Estimated scope:** Medium -## Task 9: Implement observability baseline +## Task 9: Bootstrap and accept MVP server -**Description:** Add bounded structured logging, private metrics/exporters/dashboards, release correlation, and symptom alerts/runbook links. +**Description:** Follow the reviewed bootstrap runbook, deploy manually, run one-time data/demo setup, and capture MVP evidence. **Acceptance criteria:** -- [ ] All required RED/USE/worker/SSE/backup/TLS signals exist with bounded labels. -- [ ] Logs rotate and contain request/release correlation but no forbidden fields. -- [ ] Page/ticket alerts are test-fired and monitoring is not public. +- [ ] Access/firewall/Docker/DNS/TLS/deployment boundaries pass. +- [ ] Three-hash release, migration, import, accounts, tenant isolation, worker/SSE, and font pass. +- [ ] MVP manual health/log/disk/certificate/queue checks and encrypted backup/restore pass. -**Verification:** Induced staging failure located from telemetry, cardinality/redaction checks, alert delivery test. +**Verification:** Completed bootstrap/acceptance record. -**Dependencies:** Checkpoint A and approved monitoring versions/budgets +**Dependencies:** Tasks 6–8 and approved server/window -**Files likely touched:** Compose monitoring slice/config/dashboards/tests, split to <=5 files per commit - -**Estimated scope:** Split into multiple Medium tasks - -## Task 10: Implement encrypted backup and retention - -**Description:** Produce verified logical PostgreSQL backups, encrypt before durable storage, upload/verify remotely, retain three local/seven remote, and emit metrics. - -**Acceptance criteria:** -- [ ] Archive/listing/checksums/manifest/remote verification gate success. -- [ ] Backup contains none of the explicit exclusions. -- [ ] Cleanup never removes the last known-good copy and uses exact validated paths. - -**Verification:** Backup tests against fixture DB/object store, archive exclusion inspection, failed-upload/cleanup tests. - -**Dependencies:** Tasks 1–3 and approved encryption/provider decisions - -**Files likely touched:** backup script/config/tests/docs (maximum 5 files per slice) - -**Estimated scope:** Medium (3–5 files) - -## Task 11: Implement guarded restore and rehearsal - -**Description:** Validate/decrypt/trust-check an explicit backup, require all production guards, restore into an empty destination, and run application validation. - -**Acceptance criteria:** -- [ ] Each absent/mismatched production factor aborts before mutation. -- [ ] Partial/failed restore is never treated as usable or blindly retried. -- [ ] Clean rehearsal passes tenant/session/cold-cache/worker/SSE checks within two hours. - -**Verification:** Destructive tests only against exact disposable fixtures, production-guard negative matrix, timed rehearsal. - -**Dependencies:** Task 10 and approved switch strategy - -**Files likely touched:** restore script/tests/docs (maximum 5 files per slice) - -**Estimated scope:** Medium (3–5 files) - -## Checkpoint B: Release and recovery - -- [ ] Tasks 6–11 pass in staging with release/rollback/backup/restore evidence. -- [ ] Registry, runner, monitoring, object storage, and secrets remain least privilege/private. - -## Task 12: Bootstrap and accept first production-shaped server - -**Description:** Follow the reviewed bootstrap runbook, deploy an approved manifest, run the one-time data/demo setup, and capture every acceptance criterion. - -**Acceptance criteria:** -- [ ] Access/firewall/Docker/DNS/TLS/deploy/monitoring boundaries pass. -- [ ] Migration/import/demo accounts/tenant isolation/analytics SSE pass. -- [ ] Encrypted remote backup and clean restore rehearsal meet RPO/RTO. - -**Verification:** Completed runbook evidence and stakeholder acceptance review. - -**Dependencies:** Checkpoint B and approved host/change window - -**Files likely touched:** evidence/release records only; no secret/raw runtime artifacts +**Files likely touched:** non-sensitive release/evidence records only **Estimated scope:** Operational session -## Final checkpoint +## Checkpoint B: MVP ready -- [ ] Every measurable acceptance criterion in the approved spec passes. -- [ ] Active and previous releases are exact and rollback-ready. -- [ ] Remaining decisions are resolved or explicitly defer launch. -- [ ] Documentation reflects implemented reality and no implementation gate was bypassed. +- [ ] Every MVP acceptance criterion passes. +- [ ] Active/previous three-hash releases and rollback are ready. +- [ ] Clean restore meets RPO/RTO. +- [ ] Stakeholder explicitly confirms the core project is ready, opening the observability phase. + +## Task 10: Add Prometheus and private exporters + +**Description:** Add post-MVP collection for application, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS signals. + +**Acceptance criteria:** +- [ ] Required signals appear with bounded labels and retention. +- [ ] Prometheus/exporters are private and resource-limited. +- [ ] Merchant/user/request/card identifiers are absent from metric labels. + +**Verification:** Target health, signal/cardinality/resource tests. + +**Dependencies:** Checkpoint B + +**Files likely touched:** split into <=5-file monitoring slices + +**Estimated scope:** Multiple Medium slices + +## Task 11: Add Grafana dashboards and alerts + +**Description:** Add private dashboards, release correlation, actionable symptom alerts, runbook links, and delivery testing. + +**Acceptance criteria:** +- [ ] Service, worker/SSE, data, host, backup, and TLS dashboards answer the spec's operator questions. +- [ ] Only page/ticket alerts exist and each is actionable. +- [ ] Induced failures alert and can be diagnosed without source archaeology. + +**Verification:** Dashboard review, induced failure, alert test-fire, redaction check. + +**Dependencies:** Task 10 + +**Files likely touched:** split into <=5-file dashboard/alert slices + +**Estimated scope:** Multiple Medium slices + +## Final observability checkpoint before AI + +- [ ] Prometheus/Grafana phase meets its acceptance criteria. +- [ ] Alert delivery and runbook ownership are proven. +- [ ] Stakeholder explicitly approves beginning AI implementation. +- [ ] No CI/CD pipeline or automated deployment system was introduced.