starcoach-hackathon/zarinpal-analytics-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: replan manual platform delivery

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 18:35:36 +03:30
parent 789f60e2ca
commit ec7a7753f6
2 changed files with 215 additions and 244 deletions
Download Patch File Download Diff File Expand all files Collapse all files

172
tasks/plan.md
View File

@@ -1,139 +1,125 @@
# Implementation Plan: Platform Operations # Implementation Plan: Platform Operations
- Status: Planned from draft specification; implementation blocked pending explicit approval - Status: Planned from draft revision; implementation blocked pending explicit spec approval
- Capability: Platform operations - Capability: Platform operations
- Task list target: `tasks/todo.md` - Task list target: `tasks/todo.md`
- Last updated: 2026-08-20 - Last updated: 2026-08-20
## Overview ## Confirmed implementation direction
Implement the approved Platform Operations contract as a single-server, production-shaped Docker Compose platform for `zarinpal.amiirkhl.ir`. The deployment repository remains the only operational coordinator; backend and frontend source stay in their own repositories and publish immutable images. Work proceeds from decisions and validation contracts through topology, edge, lifecycle, release automation, recovery, and end-to-end proof. Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds.
No task below is authorized merely because it appears in this plan. The Platform Operations spec must be approved and its blocking decisions resolved first. There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work.
No task below is authorized merely because it appears here.
## Dependency graph ## Dependency graph
```text ```text
approve spec + resolve platform versions/host/runner/backup decisions approve revised spec + resolve server/runtime contracts
| |
+--> release manifest + configuration/secret schemas +--> safe config + manual release-record contract
| | | |
| +--> Compose networks/data services | +--> Compose data/app services
| | | | | |
| | +--> migration/import/admin lifecycle | | +--> migrate/import/admin separation
| | +--> API/worker/frontend services | | +--> manual deploy/rollback
| | | |
| +--> image/build contracts in service repositories | +--> Nginx/Certbot routing and TLS
| |
| +--> cross-repository candidate events
| |
+--> Nginx/TLS/routing contract implementation +--> Qlockify backup adaptation --> guarded restore --> rehearsal
| |
+--> observability baseline +--> first-server bootstrap --> MVP acceptance
| |
+--> backup --> guarded restore --> restore rehearsal v
| Prometheus/Grafana phase
+--> host bootstrap validation |
| v
v AI may begin
full release/rollback/DR acceptance
``` ```
## Architecture decisions ## Architecture decisions
- Preserve ADR 0001's three-repository topology. - Preserve ADR 0001's three independent repositories and ignored nested service checkouts.
- Build application images once in service CI and deploy immutable digests through ADR 0002 release manifests. - Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records.
- Consume the licensed font/license through the ordinary frontend context per accepted ADR 0003 and frontend commit `64f5ac2`; no deployment injection path. - Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license.
- Keep PostgreSQL authoritative and backed up. Redis uses routine AOF persistence but remains excluded from disaster-recovery backup. - Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags.
- Separate API, worker, migration, one-time import, and administrative commands. No recurring CSV or scheduled application worker exists. - Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup.
- Use expand/contract migrations and application-image rollback; never automatic database down migrations. - Separate API, worker, migration, one-time CSV import, and administrative processes.
- Keep Nginx as the only public entry point and monitoring/operator access private. - Keep Asynq pinned to pre-1.0 `v0.26.0`.
- Use automatic Let's Encrypt/Certbot TLS.
- Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI.
## Phases and checkpoints ## Phase 0: approve exact contracts
### Phase 0: approval and evidence - Approve the revised Platform Operations spec.
- Resolve server/provider/capacity/user/path, image tags, resource limits, repository URLs/commands/health routes, backup endpoint/credentials/schedule, Redis AOF choice, release-record format, operator/secret escrow, and MVP-ready/observability gates.
- Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery.
- Obtain the frontend build/static health contract.
- Resolve host/provider/capacity, container privilege model, forge/runner/registry versions, exact image versions/digests, resource budgets, API health/routes, backup encryption/provider, certificate-renewal method, and operator ownership. Checkpoint: no implementation file exists before explicit approval and resolved inputs.
- Approve `docs/PLATFORM-OPERATIONS-SPEC.md` and ADR 0002. ADR 0003 is already accepted.
- Obtain approved machine-readable backend/frontend interface and runtime command contracts.
Checkpoint: no implementation file exists before the approval record and decisions are complete. ## Phase 1: configuration and Compose foundation
### Phase 1: release and configuration foundation - Add safe tracked configuration examples and validate protected external secret paths.
- Define non-secret active/previous release records containing the three SHAs and health/migration evidence.
- Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles.
- Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts.
- Define and validate the release-manifest schema, active/previous release records, exact three-repository/image provenance, and compatibility fields. Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin.
- Define tracked safe configuration examples plus the external host secret-file contract.
- Add policy tests that reject floating images, unintended ports, secret-like values, missing health checks, or font injection paths.
Checkpoint: a synthetic manifest/config validates offline and contains no secret values or mutable references. ## Phase 2: Nginx, TLS, and lifecycle
### Phase 2: runtime topology and edge - Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`.
- Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior.
- Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset.
- Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release.
- Implement isolated Compose networks, named volumes, PostgreSQL/Redis health/resources, private application services, and monitoring profile. Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes.
- Implement one-off migration, import, and administrative profiles without coupling them to normal startup.
- Implement canonical-domain Nginx/TLS, SPA/API separation, exact SSE route behavior, security headers, limits, and health routing.
- Add focused topology/routing tests before deployment automation.
Checkpoint: local/integration inspection proves only Nginx publishes 80/443, startup gates are correct, and SSE is not buffered. ## Phase 3: backup, restore, and first-server acceptance
### Phase 3: service artifact and lifecycle coordination - Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials.
- Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only.
- Add exact destination and production confirmation guards.
- Bootstrap the server, deploy the MVP, perform one-time import/credential setup, create an encrypted remote backup, and rehearse clean restore within two hours.
- Coordinate backend image commands/health/labels and frontend static/font health with the owning repositories through approved changes there; do not duplicate their source here. Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks.
- Implement deployment locks, preflight, migration, service replacement, postflight, observation, and application rollback using release manifests.
- Implement Redis-loss reconciliation acceptance with backend/worker owners, preserving Asynq `v0.26.0` as a pinned pre-1.0 dependency.
Checkpoint: an induced migration/readiness/font/worker failure preserves or restores the prior release. ## Phase 4: required observability before AI
### Phase 4: CI/CD and provenance - Add Prometheus, Grafana, and minimum private exporters/instrumentation.
- Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards.
- Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery.
- Validate the observability acceptance gate before any AI implementation starts.
- Add deployment-repository validation and production-release workflows. Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator.
- Coordinate service-repository workflows that publish immutable images and send candidate events only after required checks.
- Enforce protected environment approval, serialized release, registry digest retention, SBOM/provenance/vulnerability gates, and secret isolation.
Checkpoint: PRs never deploy; backend/frontend candidate events cannot directly mutate production; one manifest controls a tested release and rollback.
### Phase 5: observability and recovery
- Implement bounded structured container logging, Prometheus/exporters/Grafana, dashboards, and symptom alerts with runbook links.
- Implement encrypted logical PostgreSQL backup, remote verification, three-local/seven-remote retention, and telemetry.
- Implement exact-destination guarded restore into an empty destination and application validation.
- Rehearse restore and record achieved RPO/RTO.
Checkpoint: induced failures are diagnosable from telemetry, backup age alert fires, and clean restore succeeds within two hours.
### Phase 6: bootstrap and full acceptance
- Convert the reviewed first-server checklist into only the minimum approved automation, retaining manual safety/identity gates.
- Bootstrap the selected host, stage DNS/TLS, deploy a release, run one-time import/credentials, and complete alert/backup/restore handoff.
- Execute every measurable criterion from the Platform Operations spec and retain non-sensitive evidence.
Checkpoint: stakeholder review confirms release, rollback, Redis-loss, backup/restore, exposure, SLO, and operator-boundary evidence.
## Risks and mitigations ## Risks and mitigations
| Risk | Impact | Mitigation | | Risk | Impact | Mitigation |
|---|---|---| |---|---|---|
| Single-host failure | Full outage | Immutable rebuild path, encrypted off-host PostgreSQL backup, two-hour restore rehearsal, documented DNS/host replacement | | Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record |
| Cross-repository race | Incompatible release | Candidate events only, serialized deployment lock, exact manifest, compatibility metadata, one coordinator | | Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` |
| Migration blocks rollback | Extended outage/data risk | Expand/contract, measured locks, fresh verified backup, no automatic down migration | | Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time |
| Redis loss drops in-flight queue/cache | Partial analytics interruption | AOF for routine restart, PostgreSQL authority, idempotent stale-run reconciliation, cold-cache tests | | Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration |
| SSE proxy buffering/timeouts | Dashboard appears stuck | Exact route, buffering/compression/cache off, heartbeat-aware timeout, integration test | | Redis loss drops queue/cache | Partial analytics interruption | Optional AOF, PostgreSQL authority, idempotent reconciliation/cold-cache tests |
| Docker privilege compromise | Host root compromise | Approve rootless/dedicated model, restricted machine identity/SSH, no daemon TCP exposure, minimal host users | | SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test |
| Secrets leak through Compose/log/backup | Credential/data compromise | File-mounted least privilege, redaction tests, no env in backup, rotate-first incident rule | | Docker-group compromise | Host-root compromise | Dedicated key-only trusted account, restricted SSH, no daemon TCP exposure |
| Backup exists but is unusable | Data loss beyond RTO | Archive/checksum/remote verification plus monthly clean restore | | Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests |
| Font missing/divergent | Broken approved typography/license trail | Frontend-owned tracked inputs at `64f5ac2+`, build/static smoke, digest provenance, no second path | | Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal |
| Monitoring overload/noise | Lost diagnostics or ignored alerts | Bounded labels/retention, symptom alerts, test-fire, only page/ticket severities | | Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI |
## Parallelization after contracts are fixed ## Parallelization after approval
- Safe in parallel: Nginx focused tests, backup/restore tooling, observability dashboards, and CI validation, provided the shared manifest/config schemas are merged first. - Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed.
- Sequential: schema migrations, production deploy/rollback, database restore, DNS/TLS cutover, and one-time import. - Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate.
- Coordinated across repositories: backend commands/health/migration metadata, frontend image/font health, and candidate-event payload. Providers define contracts before consumers implement them. - Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced.
## Completion gate ## Completion
- Every task in `tasks/todo.md` meets acceptance and verification evidence. - Every authorized task in `tasks/todo.md` passes its acceptance and verification.
- All spec acceptance criteria pass on a production-shaped host. - The active stack maps to exact deployment/backend/frontend commits.
- No unresolved decision remains hidden in a default or script. - Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass.
- Active/previous release manifests, SLO dashboards, alerts, backup, restore rehearsal, and operator handoff are complete. - Prometheus/Grafana acceptance passes before AI begins.
- Backend/frontend repositories contain only their approved coordinated changes; the deployment repository contains no application source, raw data, secrets, credentials, TLS keys, backups, database volumes, or duplicate font/license files. - Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets.

287
tasks/todo.md
View File

@@ -1,243 +1,228 @@
# Platform Operations implementation tasks # Platform Operations implementation tasks
- Status: Not authorized — wait for explicit approval of the draft spec - Status: Not authorized — wait for explicit approval of the revised spec
- Canonical plan: `tasks/plan.md` - Canonical plan: `tasks/plan.md`
## Task 0: Approve specification and resolve blocking decisions ## Task 0: Approve revision and resolve implementation inputs
**Description:** Review the platform spec/ADRs and record every host, runner, version, route, capacity, backup, TLS, and ownership decision before implementation. **Description:** Approve the confirmed manual Qlockify-style intent and resolve every remaining server/runtime/backup/phase-gate decision.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] `PLATFORM-OPERATIONS-SPEC.md` status is explicitly approved by the stakeholder. - [ ] Revised Platform Operations spec is explicitly approved.
- [ ] ADR 0002 is accepted or replaced; ADR 0003 remains consistent with frontend commit `64f5ac2`. - [ ] ADRs 0001–0003 match the approved topology, manual release, and font decisions.
- [ ] All remaining platform decisions have named answers/evidence; the frontend font ownership/delivery decision is already resolved by `64f5ac2` and ADR 0003. - [ ] Every remaining decision has a named answer/evidence.
**Verification:** Manual stakeholder approval and clean documentation diff. **Verification:** Stakeholder approval plus clean documentation review.
**Dependencies:** None **Dependencies:** None
**Files likely touched:** `docs/PLATFORM-OPERATIONS-SPEC.md`, `docs/decisions/0002-immutable-cross-repository-releases.md` **Files likely touched:** spec and ADRs only
**Estimated scope:** Small (1–2 files) **Estimated scope:** Small
## Task 1: Define release manifest and safe configuration schemas ## Task 1: Define safe configuration and release records
**Description:** Create machine-validated schemas/examples for exact release provenance, compatibility metadata, active/previous pointers, safe public configuration, and external secret files. **Description:** Define tracked placeholders, external protected secrets, deployment locking, and non-secret active/previous release records.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Synthetic valid/invalid manifests prove immutable three-repository/image constraints. - [ ] Release record contains three SHAs, local image ids, migration, backup, operator, and health evidence.
- [ ] Examples contain no secret-like values and name every required configuration input. - [ ] Configuration examples contain no real or plausible secret values.
- [ ] Font provenance is only frontend SHA/image digest; no injection field/path exists. - [ ] Font provenance uses frontend commit/build only; no second asset path exists.
**Verification:** Schema tests plus secret scan over staged files. **Verification:** Schema/format tests, secret scan, invalid-record tests.
**Dependencies:** Task 0 **Dependencies:** Task 0
**Files likely touched:** release/config schemas, safe examples, focused tests (maximum 5 files) **Files likely touched:** examples/schema/tests/docs, maximum 5 files
**Estimated scope:** Medium (3–5 files) **Estimated scope:** Medium
## Task 2: Implement data-service Compose foundation ## Task 2: Implement PostgreSQL and Redis Compose foundation
**Description:** Add PostgreSQL/Redis pinned services, isolated data network, volumes, authentication, health checks, AOF policy, resources, and no host port publishing. **Description:** Add pinned PostgreSQL 18 and Redis 7 Alpine, private data network, volumes, authentication, health, resources, and approved Redis AOF choice.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Compose renders with pinned digests and healthy PostgreSQL/Redis. - [ ] Both services become healthy with explicit limits/tags.
- [ ] Redis loss is explicitly non-authoritative and its volume is excluded from backup. - [ ] Neither publishes a host port.
- [ ] External/local port inspection finds no PostgreSQL/Redis host listener. - [ ] Redis is excluded from backup and loss does not affect PostgreSQL authority.
**Verification:** Compose validation, health inspection, network/port policy tests. **Verification:** Compose validation, health, port/network, volume, Redis-loss tests.
**Dependencies:** Task 1 **Dependencies:** Task 1
**Files likely touched:** Compose files and focused topology tests (maximum 5 files) **Files likely touched:** Compose/config/tests, maximum 5 files
**Estimated scope:** Medium (3–5 files) **Estimated scope:** Medium
## Task 3: Implement API, worker, and one-off lifecycle boundaries ## Task 3: Implement backend, worker, and one-off boundaries
**Description:** Wire immutable backend API/worker plus migration/import/admin one-offs with health gates, profiles, locks, graceful stop, and validated configuration. **Description:** Wire API/worker plus separate migration/import/admin commands and graceful lifecycle.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Normal startup waits for migration but never runs import/admin automatically. - [ ] Migration completes before API/worker readiness.
- [ ] Worker is private/event-driven and retains pinned pre-1.0 Asynq `v0.26.0` contract. - [ ] Normal startup never runs import/admin; import uses one exact read-only path.
- [ ] Failed migration blocks candidate API/worker; import path is exact/read-only/idempotent. - [ ] Worker is private/event-driven and uses pinned pre-1.0 Asynq `v0.26.0`.
**Verification:** Lifecycle integration tests including failed migration and repeated import fixture. **Verification:** Startup/failure/import-idempotency/shutdown integration tests.
**Dependencies:** Tasks 1–2 and approved backend command/health contract **Dependencies:** Task 2 and approved backend commands
**Files likely touched:** Compose lifecycle definitions, wrapper, focused tests (maximum 5 files) **Files likely touched:** Compose/service wrappers/tests, maximum 5 files per slice
**Estimated scope:** Medium (3–5 files) **Estimated scope:** Medium
## Task 4: Implement frontend and font artifact health ## Task 4: Implement frontend build and font health
**Description:** Consume the ordinary frontend image/build context and verify static SPA plus tracked font output without copying/injecting assets in deployment. **Description:** Build the nested frontend checkout normally and verify static SPA/font output without deployment copies or injection.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Candidate maps to frontend SHA/digest and serves expected font URL/MIME. - [ ] Build uses recorded frontend commit at/after `64f5ac2`.
- [ ] Missing font/license input fails frontend CI or candidate smoke. - [ ] Expected font resource returns 200 with correct content type.
- [ ] Deployment files contain no font binary, license copy, private fetch, or font secret. - [ ] Deployment repository contains no font binary, `FontLicense.txt`, fetch, credential, or alternate path.
**Verification:** Image/static health test and repository policy scan. **Verification:** Build/static health plus repository-policy scan.
**Dependencies:** Task 1 and approved frontend image contract at/after `64f5ac2` **Dependencies:** Task 1 and approved frontend build contract
**Files likely touched:** frontend service definition and focused smoke/policy tests (maximum 5 files) **Files likely touched:** frontend deployment wrapper/service/tests, maximum 5 files
**Estimated scope:** Small/Medium (2–4 files) **Estimated scope:** Medium
## Task 5: Implement Nginx canonical TLS and routing ## Task 5: Implement Nginx and automatic TLS
**Description:** Add canonical host/default rejection, ACME/TLS, API/frontend separation, exact SSE proxy behavior, security headers, limits, and generic health routes. **Description:** Adapt Qlockify's Nginx/Certbot pattern for canonical routing, Let's Encrypt, API/frontend separation, exact SSE behavior, security headers, and health.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Only 80/443 are public; redirects/default-host behavior is exact. - [ ] Only 80/443 are public; unknown hosts reject and canonical redirect is exact.
- [ ] SSE buffering/cache/compression are off and heartbeat/replay survives reconnect. - [ ] Staging/production issuance and renewal/reload pass.
- [ ] API never falls through to SPA; security/TLS/static-cache tests pass. - [ ] SSE heartbeat/reconnect streams without buffering/cache/compression.
**Verification:** `nginx -t`, TLS/header/routing tests, streamed timing/reconnect test. **Verification:** `nginx -t`, external TLS/header/routing checks, streamed SSE test.
**Dependencies:** Tasks 3–4 and approved API/SSE paths/CSP **Dependencies:** Tasks 3–4 and approved routes/CSP
**Files likely touched:** Nginx template/entrypoint and focused tests (maximum 5 files) **Files likely touched:** Nginx/Certbot assets/tests, maximum 5 files per slice
**Estimated scope:** Medium (3–5 files) **Estimated scope:** Medium
## Checkpoint A: Runtime foundation ## Checkpoint A: Runtime MVP foundation
- [ ] Tasks 1–5 pass together from a clean Compose project. - [ ] Tasks 1–5 pass from clean nested checkouts.
- [ ] Only Nginx is public and all health/startup boundaries behave as specified. - [ ] Only Nginx is public; migration/import/admin/font/SSE boundaries behave correctly.
- [ ] No secret, raw data, backup, TLS key, or duplicate font/license asset is tracked. - [ ] No secret/raw data/runtime/font duplication enters deployment Git.
## Task 6: Implement release preflight, deploy, and rollback ## Task 6: Adapt Qlockify manual deploy and rollback
**Description:** Build the serialized manifest-driven operator workflow for preflight, migration, service replacement, postflight, observation, active/previous switch, and rollback. **Description:** Modify Qlockify's deploy flow for ZarinPal paths/services and three-hash release records while removing unconditional destructive Git synchronization.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Deploy uses image digests only and records evidence without secrets. - [ ] Latest `origin/main` is fast-forwarded and three hashes recorded before build.
- [ ] Candidate health/migration/font failure preserves/restores prior release. - [ ] Dirty/diverged checkout, pull, build, migration, or health failure preserves the active release.
- [ ] Application-only rollback completes within 15 minutes and never auto-runs down migration. - [ ] Previous hashes rebuild/restore within 15 minutes without automatic down migration.
**Verification:** Shell/unit tests plus induced staging failures and timed rollback drill. **Verification:** Script tests plus induced staging failures and timed rollback drill.
**Dependencies:** Checkpoint A **Dependencies:** Checkpoint A
**Files likely touched:** deploy/release scripts and focused tests (maximum 5 files per incremental slice) **Files likely touched:** deploy script/release helper/tests/docs, maximum 5 files per slice
**Estimated scope:** Split into multiple Medium tasks during implementation **Estimated scope:** Medium
## Task 7: Implement deployment-repository CI/CD ## Task 7: Adapt Qlockify encrypted backup/upload
**Description:** Validate docs/config/Compose/Nginx/scripts/policies, accept authenticated candidates, enforce protected approval/locking, and invoke the manifest-driven release. **Description:** Reuse Qlockify's PostgreSQL dump, OpenSSL, rclone upload, and retention flow with ZarinPal paths/services/credentials and a non-secret manifest.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] PR workflows validate only and cannot access production secrets/deploy. - [ ] Archive/listing/checksums/encryption/remote verification gate success.
- [ ] Production release requires gates plus human approval and is serialized. - [ ] Exactly three verified encrypted local and seven remote backups are retained.
- [ ] Provenance/SBOM/scan/secret results attach to the release record. - [ ] `.env`, secrets, Redis, TLS, CSV, credentials, logs, monitoring, font, and license are excluded.
**Verification:** Forge event tests with denied PR deploy, concurrent candidate events, failed gate, and approved release. **Verification:** Fixture database/object-store tests, exclusion inspection, failed-upload/cleanup tests.
**Dependencies:** Task 6 and confirmed forge/runner/registry versions **Dependencies:** Tasks 1–3 and approved backup inputs
**Files likely touched:** workflow plus CI-focused helpers/tests (maximum 5 files) **Files likely touched:** adapted backup/upload scripts/tests/docs, maximum 5 files per slice
**Estimated scope:** Medium (3–5 files) **Estimated scope:** Medium
## Task 8: Coordinate backend/frontend candidate workflows ## Task 8: Adapt Qlockify guarded restore
**Description:** In separately approved service-repository changes, publish one immutable tested image and send candidate metadata without direct production SSH. **Description:** Reuse Qlockify download/decrypt/restore flow while requiring explicit backup id, exact destination, empty target, and all production guards.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] Backend gates include Go/migration/import/Asynq queue/recovery verification. - [ ] Every absent/mismatched production factor aborts before mutation.
- [ ] Frontend gates include Svelte/accessibility/RTL/build/font verification. - [ ] No environment/secret/Redis restoration occurs.
- [ ] Service events cannot mutate production and use least-privilege credentials. - [ ] Clean restore passes schema/count/tenant/login/cold-cache/worker/SSE checks within two hours.
**Verification:** Each repository's CI on PR/main plus deployment candidate receipt test. **Verification:** Disposable-database tests, guard negative matrix, timed rehearsal.
**Dependencies:** Task 7 and separate authorization in each service repository **Dependencies:** Task 7
**Files likely touched:** owned service workflow/tests only; not this deployment task **Files likely touched:** adapted restore scripts/tests/docs, maximum 5 files per slice
**Estimated scope:** Separate cross-repository tasks **Estimated scope:** Medium
## Task 9: Implement observability baseline ## Task 9: Bootstrap and accept MVP server
**Description:** Add bounded structured logging, private metrics/exporters/dashboards, release correlation, and symptom alerts/runbook links. **Description:** Follow the reviewed bootstrap runbook, deploy manually, run one-time data/demo setup, and capture MVP evidence.
**Acceptance criteria:** **Acceptance criteria:**
- [ ] All required RED/USE/worker/SSE/backup/TLS signals exist with bounded labels. - [ ] Access/firewall/Docker/DNS/TLS/deployment boundaries pass.
- [ ] Logs rotate and contain request/release correlation but no forbidden fields. - [ ] Three-hash release, migration, import, accounts, tenant isolation, worker/SSE, and font pass.
- [ ] Page/ticket alerts are test-fired and monitoring is not public. - [ ] MVP manual health/log/disk/certificate/queue checks and encrypted backup/restore pass.
**Verification:** Induced staging failure located from telemetry, cardinality/redaction checks, alert delivery test. **Verification:** Completed bootstrap/acceptance record.
**Dependencies:** Checkpoint A and approved monitoring versions/budgets **Dependencies:** Tasks 6–8 and approved server/window
**Files likely touched:** Compose monitoring slice/config/dashboards/tests, split to <=5 files per commit **Files likely touched:** non-sensitive release/evidence records only
**Estimated scope:** Split into multiple Medium tasks
## Task 10: Implement encrypted backup and retention
**Description:** Produce verified logical PostgreSQL backups, encrypt before durable storage, upload/verify remotely, retain three local/seven remote, and emit metrics.
**Acceptance criteria:**
- [ ] Archive/listing/checksums/manifest/remote verification gate success.
- [ ] Backup contains none of the explicit exclusions.
- [ ] Cleanup never removes the last known-good copy and uses exact validated paths.
**Verification:** Backup tests against fixture DB/object store, archive exclusion inspection, failed-upload/cleanup tests.
**Dependencies:** Tasks 1–3 and approved encryption/provider decisions
**Files likely touched:** backup script/config/tests/docs (maximum 5 files per slice)
**Estimated scope:** Medium (3–5 files)
## Task 11: Implement guarded restore and rehearsal
**Description:** Validate/decrypt/trust-check an explicit backup, require all production guards, restore into an empty destination, and run application validation.
**Acceptance criteria:**
- [ ] Each absent/mismatched production factor aborts before mutation.
- [ ] Partial/failed restore is never treated as usable or blindly retried.
- [ ] Clean rehearsal passes tenant/session/cold-cache/worker/SSE checks within two hours.
**Verification:** Destructive tests only against exact disposable fixtures, production-guard negative matrix, timed rehearsal.
**Dependencies:** Task 10 and approved switch strategy
**Files likely touched:** restore script/tests/docs (maximum 5 files per slice)
**Estimated scope:** Medium (3–5 files)
## Checkpoint B: Release and recovery
- [ ] Tasks 6–11 pass in staging with release/rollback/backup/restore evidence.
- [ ] Registry, runner, monitoring, object storage, and secrets remain least privilege/private.
## Task 12: Bootstrap and accept first production-shaped server
**Description:** Follow the reviewed bootstrap runbook, deploy an approved manifest, run the one-time data/demo setup, and capture every acceptance criterion.
**Acceptance criteria:**
- [ ] Access/firewall/Docker/DNS/TLS/deploy/monitoring boundaries pass.
- [ ] Migration/import/demo accounts/tenant isolation/analytics SSE pass.
- [ ] Encrypted remote backup and clean restore rehearsal meet RPO/RTO.
**Verification:** Completed runbook evidence and stakeholder acceptance review.
**Dependencies:** Checkpoint B and approved host/change window
**Files likely touched:** evidence/release records only; no secret/raw runtime artifacts
**Estimated scope:** Operational session **Estimated scope:** Operational session
## Final checkpoint ## Checkpoint B: MVP ready
- [ ] Every measurable acceptance criterion in the approved spec passes. - [ ] Every MVP acceptance criterion passes.
- [ ] Active and previous releases are exact and rollback-ready. - [ ] Active/previous three-hash releases and rollback are ready.
- [ ] Remaining decisions are resolved or explicitly defer launch. - [ ] Clean restore meets RPO/RTO.
- [ ] Documentation reflects implemented reality and no implementation gate was bypassed. - [ ] Stakeholder explicitly confirms the core project is ready, opening the observability phase.
## Task 10: Add Prometheus and private exporters
**Description:** Add post-MVP collection for application, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS signals.
**Acceptance criteria:**
- [ ] Required signals appear with bounded labels and retention.
- [ ] Prometheus/exporters are private and resource-limited.
- [ ] Merchant/user/request/card identifiers are absent from metric labels.
**Verification:** Target health, signal/cardinality/resource tests.
**Dependencies:** Checkpoint B
**Files likely touched:** split into <=5-file monitoring slices
**Estimated scope:** Multiple Medium slices
## Task 11: Add Grafana dashboards and alerts
**Description:** Add private dashboards, release correlation, actionable symptom alerts, runbook links, and delivery testing.
**Acceptance criteria:**
- [ ] Service, worker/SSE, data, host, backup, and TLS dashboards answer the spec's operator questions.
- [ ] Only page/ticket alerts exist and each is actionable.
- [ ] Induced failures alert and can be diagnosed without source archaeology.
**Verification:** Dashboard review, induced failure, alert test-fire, redaction check.
**Dependencies:** Task 10
**Files likely touched:** split into <=5-file dashboard/alert slices
**Estimated scope:** Multiple Medium slices
## Final observability checkpoint before AI
- [ ] Prometheus/Grafana phase meets its acceptance criteria.
- [ ] Alert delivery and runbook ownership are proven.
- [ ] Stakeholder explicitly approves beginning AI implementation.
- [ ] No CI/CD pipeline or automated deployment system was introduced.