starcoach-hackathon/zarinpal-analytics-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: replan manual platform delivery

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 18:35:36 +03:30
parent 789f60e2ca
commit ec7a7753f6
2 changed files with 215 additions and 244 deletions
Download Patch File Download Diff File Expand all files Collapse all files

172
tasks/plan.md
View File

@@ -1,139 +1,125 @@
# Implementation Plan: Platform Operations
- Status: Planned from draft specification; implementation blocked pending explicit approval
- Status: Planned from draft revision; implementation blocked pending explicit spec approval
- Capability: Platform operations
- Task list target: `tasks/todo.md`
- Last updated: 2026-08-20
## Overview
## Confirmed implementation direction
Implement the approved Platform Operations contract as a single-server, production-shaped Docker Compose platform for `zarinpal.amiirkhl.ir`. The deployment repository remains the only operational coordinator; backend and frontend source stay in their own repositories and publish immutable images. Work proceeds from decisions and validation contracts through topology, edge, lifecycle, release automation, recovery, and end-to-end proof.
Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds.
No task below is authorized merely because it appears in this plan. The Platform Operations spec must be approved and its blocking decisions resolved first.
There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work.
No task below is authorized merely because it appears here.
## Dependency graph
```text
approve spec + resolve platform versions/host/runner/backup decisions
approve revised spec + resolve server/runtime contracts
|
+--> release manifest + configuration/secret schemas
+--> safe config + manual release-record contract
| |
| +--> Compose networks/data services
| +--> Compose data/app services
| | |
| | +--> migration/import/admin lifecycle
| | +--> API/worker/frontend services
| | +--> migrate/import/admin separation
| | +--> manual deploy/rollback
| |
| +--> image/build contracts in service repositories
| |
| +--> cross-repository candidate events
| +--> Nginx/Certbot routing and TLS
|
+--> Nginx/TLS/routing contract implementation
+--> Qlockify backup adaptation --> guarded restore --> rehearsal
|
+--> observability baseline
|
+--> backup --> guarded restore --> restore rehearsal
|
+--> host bootstrap validation
|
v
full release/rollback/DR acceptance
+--> first-server bootstrap --> MVP acceptance
|
v
Prometheus/Grafana phase
|
v
AI may begin
```
## Architecture decisions
- Preserve ADR 0001's three-repository topology.
- Build application images once in service CI and deploy immutable digests through ADR 0002 release manifests.
- Consume the licensed font/license through the ordinary frontend context per accepted ADR 0003 and frontend commit `64f5ac2`; no deployment injection path.
- Keep PostgreSQL authoritative and backed up. Redis uses routine AOF persistence but remains excluded from disaster-recovery backup.
- Separate API, worker, migration, one-time import, and administrative commands. No recurring CSV or scheduled application worker exists.
- Use expand/contract migrations and application-image rollback; never automatic database down migrations.
- Keep Nginx as the only public entry point and monitoring/operator access private.
- Preserve ADR 0001's three independent repositories and ignored nested service checkouts.
- Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records.
- Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license.
- Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags.
- Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup.
- Separate API, worker, migration, one-time CSV import, and administrative processes.
- Keep Asynq pinned to pre-1.0 `v0.26.0`.
- Use automatic Let's Encrypt/Certbot TLS.
- Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI.
## Phases and checkpoints
## Phase 0: approve exact contracts
### Phase 0: approval and evidence
- Approve the revised Platform Operations spec.
- Resolve server/provider/capacity/user/path, image tags, resource limits, repository URLs/commands/health routes, backup endpoint/credentials/schedule, Redis AOF choice, release-record format, operator/secret escrow, and MVP-ready/observability gates.
- Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery.
- Obtain the frontend build/static health contract.
- Resolve host/provider/capacity, container privilege model, forge/runner/registry versions, exact image versions/digests, resource budgets, API health/routes, backup encryption/provider, certificate-renewal method, and operator ownership.
- Approve `docs/PLATFORM-OPERATIONS-SPEC.md` and ADR 0002. ADR 0003 is already accepted.
- Obtain approved machine-readable backend/frontend interface and runtime command contracts.
Checkpoint: no implementation file exists before explicit approval and resolved inputs.
Checkpoint: no implementation file exists before the approval record and decisions are complete.
## Phase 1: configuration and Compose foundation
### Phase 1: release and configuration foundation
- Add safe tracked configuration examples and validate protected external secret paths.
- Define non-secret active/previous release records containing the three SHAs and health/migration evidence.
- Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles.
- Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts.
- Define and validate the release-manifest schema, active/previous release records, exact three-repository/image provenance, and compatibility fields.
- Define tracked safe configuration examples plus the external host secret-file contract.
- Add policy tests that reject floating images, unintended ports, secret-like values, missing health checks, or font injection paths.
Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin.
Checkpoint: a synthetic manifest/config validates offline and contains no secret values or mutable references.
## Phase 2: Nginx, TLS, and lifecycle
### Phase 2: runtime topology and edge
- Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`.
- Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior.
- Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset.
- Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release.
- Implement isolated Compose networks, named volumes, PostgreSQL/Redis health/resources, private application services, and monitoring profile.
- Implement one-off migration, import, and administrative profiles without coupling them to normal startup.
- Implement canonical-domain Nginx/TLS, SPA/API separation, exact SSE route behavior, security headers, limits, and health routing.
- Add focused topology/routing tests before deployment automation.
Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes.
Checkpoint: local/integration inspection proves only Nginx publishes 80/443, startup gates are correct, and SSE is not buffered.
## Phase 3: backup, restore, and first-server acceptance
### Phase 3: service artifact and lifecycle coordination
- Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials.
- Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only.
- Add exact destination and production confirmation guards.
- Bootstrap the server, deploy the MVP, perform one-time import/credential setup, create an encrypted remote backup, and rehearse clean restore within two hours.
- Coordinate backend image commands/health/labels and frontend static/font health with the owning repositories through approved changes there; do not duplicate their source here.
- Implement deployment locks, preflight, migration, service replacement, postflight, observation, and application rollback using release manifests.
- Implement Redis-loss reconciliation acceptance with backend/worker owners, preserving Asynq `v0.26.0` as a pinned pre-1.0 dependency.
Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks.
Checkpoint: an induced migration/readiness/font/worker failure preserves or restores the prior release.
## Phase 4: required observability before AI
### Phase 4: CI/CD and provenance
- Add Prometheus, Grafana, and minimum private exporters/instrumentation.
- Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards.
- Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery.
- Validate the observability acceptance gate before any AI implementation starts.
- Add deployment-repository validation and production-release workflows.
- Coordinate service-repository workflows that publish immutable images and send candidate events only after required checks.
- Enforce protected environment approval, serialized release, registry digest retention, SBOM/provenance/vulnerability gates, and secret isolation.
Checkpoint: PRs never deploy; backend/frontend candidate events cannot directly mutate production; one manifest controls a tested release and rollback.
### Phase 5: observability and recovery
- Implement bounded structured container logging, Prometheus/exporters/Grafana, dashboards, and symptom alerts with runbook links.
- Implement encrypted logical PostgreSQL backup, remote verification, three-local/seven-remote retention, and telemetry.
- Implement exact-destination guarded restore into an empty destination and application validation.
- Rehearse restore and record achieved RPO/RTO.
Checkpoint: induced failures are diagnosable from telemetry, backup age alert fires, and clean restore succeeds within two hours.
### Phase 6: bootstrap and full acceptance
- Convert the reviewed first-server checklist into only the minimum approved automation, retaining manual safety/identity gates.
- Bootstrap the selected host, stage DNS/TLS, deploy a release, run one-time import/credentials, and complete alert/backup/restore handoff.
- Execute every measurable criterion from the Platform Operations spec and retain non-sensitive evidence.
Checkpoint: stakeholder review confirms release, rollback, Redis-loss, backup/restore, exposure, SLO, and operator-boundary evidence.
Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator.
## Risks and mitigations
| Risk | Impact | Mitigation |
|---|---|---|
| Single-host failure | Full outage | Immutable rebuild path, encrypted off-host PostgreSQL backup, two-hour restore rehearsal, documented DNS/host replacement |
| Cross-repository race | Incompatible release | Candidate events only, serialized deployment lock, exact manifest, compatibility metadata, one coordinator |
| Migration blocks rollback | Extended outage/data risk | Expand/contract, measured locks, fresh verified backup, no automatic down migration |
| Redis loss drops in-flight queue/cache | Partial analytics interruption | AOF for routine restart, PostgreSQL authority, idempotent stale-run reconciliation, cold-cache tests |
| SSE proxy buffering/timeouts | Dashboard appears stuck | Exact route, buffering/compression/cache off, heartbeat-aware timeout, integration test |
| Docker privilege compromise | Host root compromise | Approve rootless/dedicated model, restricted machine identity/SSH, no daemon TCP exposure, minimal host users |
| Secrets leak through Compose/log/backup | Credential/data compromise | File-mounted least privilege, redaction tests, no env in backup, rotate-first incident rule |
| Backup exists but is unusable | Data loss beyond RTO | Archive/checksum/remote verification plus monthly clean restore |
| Font missing/divergent | Broken approved typography/license trail | Frontend-owned tracked inputs at `64f5ac2+`, build/static smoke, digest provenance, no second path |
| Monitoring overload/noise | Lost diagnostics or ignored alerts | Bounded labels/retention, symptom alerts, test-fire, only page/ticket severities |
| Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record |
| Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` |
| Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time |
| Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration |
| Redis loss drops queue/cache | Partial analytics interruption | Optional AOF, PostgreSQL authority, idempotent reconciliation/cold-cache tests |
| SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test |
| Docker-group compromise | Host-root compromise | Dedicated key-only trusted account, restricted SSH, no daemon TCP exposure |
| Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests |
| Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal |
| Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI |
## Parallelization after contracts are fixed
## Parallelization after approval
- Safe in parallel: Nginx focused tests, backup/restore tooling, observability dashboards, and CI validation, provided the shared manifest/config schemas are merged first.
- Sequential: schema migrations, production deploy/rollback, database restore, DNS/TLS cutover, and one-time import.
- Coordinated across repositories: backend commands/health/migration metadata, frontend image/font health, and candidate-event payload. Providers define contracts before consumers implement them.
- Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed.
- Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate.
- Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced.
## Completion gate
## Completion
- Every task in `tasks/todo.md` meets acceptance and verification evidence.
- All spec acceptance criteria pass on a production-shaped host.
- No unresolved decision remains hidden in a default or script.
- Active/previous release manifests, SLO dashboards, alerts, backup, restore rehearsal, and operator handoff are complete.
- Backend/frontend repositories contain only their approved coordinated changes; the deployment repository contains no application source, raw data, secrets, credentials, TLS keys, backups, database volumes, or duplicate font/license files.
- Every authorized task in `tasks/todo.md` passes its acceptance and verification.
- The active stack maps to exact deployment/backend/frontend commits.
- Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass.
- Prometheus/Grafana acceptance passes before AI begins.
- Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets.

287
tasks/todo.md
View File

@@ -1,243 +1,228 @@
# Platform Operations implementation tasks
- Status: Not authorized — wait for explicit approval of the draft spec
- Status: Not authorized — wait for explicit approval of the revised spec
- Canonical plan: `tasks/plan.md`
## Task 0: Approve specification and resolve blocking decisions
## Task 0: Approve revision and resolve implementation inputs
**Description:** Review the platform spec/ADRs and record every host, runner, version, route, capacity, backup, TLS, and ownership decision before implementation.
**Description:** Approve the confirmed manual Qlockify-style intent and resolve every remaining server/runtime/backup/phase-gate decision.
**Acceptance criteria:**
- [ ] `PLATFORM-OPERATIONS-SPEC.md` status is explicitly approved by the stakeholder.
- [ ] ADR 0002 is accepted or replaced; ADR 0003 remains consistent with frontend commit `64f5ac2`.
- [ ] All remaining platform decisions have named answers/evidence; the frontend font ownership/delivery decision is already resolved by `64f5ac2` and ADR 0003.
- [ ] Revised Platform Operations spec is explicitly approved.
- [ ] ADRs 0001–0003 match the approved topology, manual release, and font decisions.
- [ ] Every remaining decision has a named answer/evidence.
**Verification:** Manual stakeholder approval and clean documentation diff.
**Verification:** Stakeholder approval plus clean documentation review.
**Dependencies:** None
**Files likely touched:** `docs/PLATFORM-OPERATIONS-SPEC.md`, `docs/decisions/0002-immutable-cross-repository-releases.md`
**Files likely touched:** spec and ADRs only
**Estimated scope:** Small (1–2 files)
**Estimated scope:** Small
## Task 1: Define release manifest and safe configuration schemas
## Task 1: Define safe configuration and release records
**Description:** Create machine-validated schemas/examples for exact release provenance, compatibility metadata, active/previous pointers, safe public configuration, and external secret files.
**Description:** Define tracked placeholders, external protected secrets, deployment locking, and non-secret active/previous release records.
**Acceptance criteria:**
- [ ] Synthetic valid/invalid manifests prove immutable three-repository/image constraints.
- [ ] Examples contain no secret-like values and name every required configuration input.
- [ ] Font provenance is only frontend SHA/image digest; no injection field/path exists.
- [ ] Release record contains three SHAs, local image ids, migration, backup, operator, and health evidence.
- [ ] Configuration examples contain no real or plausible secret values.
- [ ] Font provenance uses frontend commit/build only; no second asset path exists.
**Verification:** Schema tests plus secret scan over staged files.
**Verification:** Schema/format tests, secret scan, invalid-record tests.
**Dependencies:** Task 0
**Files likely touched:** release/config schemas, safe examples, focused tests (maximum 5 files)
**Files likely touched:** examples/schema/tests/docs, maximum 5 files
**Estimated scope:** Medium (3–5 files)
**Estimated scope:** Medium
## Task 2: Implement data-service Compose foundation
## Task 2: Implement PostgreSQL and Redis Compose foundation
**Description:** Add PostgreSQL/Redis pinned services, isolated data network, volumes, authentication, health checks, AOF policy, resources, and no host port publishing.
**Description:** Add pinned PostgreSQL 18 and Redis 7 Alpine, private data network, volumes, authentication, health, resources, and approved Redis AOF choice.
**Acceptance criteria:**
- [ ] Compose renders with pinned digests and healthy PostgreSQL/Redis.
- [ ] Redis loss is explicitly non-authoritative and its volume is excluded from backup.
- [ ] External/local port inspection finds no PostgreSQL/Redis host listener.
- [ ] Both services become healthy with explicit limits/tags.
- [ ] Neither publishes a host port.
- [ ] Redis is excluded from backup and loss does not affect PostgreSQL authority.
**Verification:** Compose validation, health inspection, network/port policy tests.
**Verification:** Compose validation, health, port/network, volume, Redis-loss tests.
**Dependencies:** Task 1
**Files likely touched:** Compose files and focused topology tests (maximum 5 files)
**Files likely touched:** Compose/config/tests, maximum 5 files
**Estimated scope:** Medium (3–5 files)
**Estimated scope:** Medium
## Task 3: Implement API, worker, and one-off lifecycle boundaries
## Task 3: Implement backend, worker, and one-off boundaries
**Description:** Wire immutable backend API/worker plus migration/import/admin one-offs with health gates, profiles, locks, graceful stop, and validated configuration.
**Description:** Wire API/worker plus separate migration/import/admin commands and graceful lifecycle.
**Acceptance criteria:**
- [ ] Normal startup waits for migration but never runs import/admin automatically.
- [ ] Worker is private/event-driven and retains pinned pre-1.0 Asynq `v0.26.0` contract.
- [ ] Failed migration blocks candidate API/worker; import path is exact/read-only/idempotent.
- [ ] Migration completes before API/worker readiness.
- [ ] Normal startup never runs import/admin; import uses one exact read-only path.
- [ ] Worker is private/event-driven and uses pinned pre-1.0 Asynq `v0.26.0`.
**Verification:** Lifecycle integration tests including failed migration and repeated import fixture.
**Verification:** Startup/failure/import-idempotency/shutdown integration tests.
**Dependencies:** Tasks 1–2 and approved backend command/health contract
**Dependencies:** Task 2 and approved backend commands
**Files likely touched:** Compose lifecycle definitions, wrapper, focused tests (maximum 5 files)
**Files likely touched:** Compose/service wrappers/tests, maximum 5 files per slice
**Estimated scope:** Medium (3–5 files)
**Estimated scope:** Medium
## Task 4: Implement frontend and font artifact health
## Task 4: Implement frontend build and font health
**Description:** Consume the ordinary frontend image/build context and verify static SPA plus tracked font output without copying/injecting assets in deployment.
**Description:** Build the nested frontend checkout normally and verify static SPA/font output without deployment copies or injection.
**Acceptance criteria:**
- [ ] Candidate maps to frontend SHA/digest and serves expected font URL/MIME.
- [ ] Missing font/license input fails frontend CI or candidate smoke.
- [ ] Deployment files contain no font binary, license copy, private fetch, or font secret.
- [ ] Build uses recorded frontend commit at/after `64f5ac2`.
- [ ] Expected font resource returns 200 with correct content type.
- [ ] Deployment repository contains no font binary, `FontLicense.txt`, fetch, credential, or alternate path.
**Verification:** Image/static health test and repository policy scan.
**Verification:** Build/static health plus repository-policy scan.
**Dependencies:** Task 1 and approved frontend image contract at/after `64f5ac2`
**Dependencies:** Task 1 and approved frontend build contract
**Files likely touched:** frontend service definition and focused smoke/policy tests (maximum 5 files)
**Files likely touched:** frontend deployment wrapper/service/tests, maximum 5 files
**Estimated scope:** Small/Medium (2–4 files)
**Estimated scope:** Medium
## Task 5: Implement Nginx canonical TLS and routing
## Task 5: Implement Nginx and automatic TLS
**Description:** Add canonical host/default rejection, ACME/TLS, API/frontend separation, exact SSE proxy behavior, security headers, limits, and generic health routes.
**Description:** Adapt Qlockify's Nginx/Certbot pattern for canonical routing, Let's Encrypt, API/frontend separation, exact SSE behavior, security headers, and health.
**Acceptance criteria:**
- [ ] Only 80/443 are public; redirects/default-host behavior is exact.
- [ ] SSE buffering/cache/compression are off and heartbeat/replay survives reconnect.
- [ ] API never falls through to SPA; security/TLS/static-cache tests pass.
- [ ] Only 80/443 are public; unknown hosts reject and canonical redirect is exact.
- [ ] Staging/production issuance and renewal/reload pass.
- [ ] SSE heartbeat/reconnect streams without buffering/cache/compression.
**Verification:** `nginx -t`, TLS/header/routing tests, streamed timing/reconnect test.
**Verification:** `nginx -t`, external TLS/header/routing checks, streamed SSE test.
**Dependencies:** Tasks 3–4 and approved API/SSE paths/CSP
**Dependencies:** Tasks 3–4 and approved routes/CSP
**Files likely touched:** Nginx template/entrypoint and focused tests (maximum 5 files)
**Files likely touched:** Nginx/Certbot assets/tests, maximum 5 files per slice
**Estimated scope:** Medium (3–5 files)
**Estimated scope:** Medium
## Checkpoint A: Runtime foundation
## Checkpoint A: Runtime MVP foundation
- [ ] Tasks 1–5 pass together from a clean Compose project.
- [ ] Only Nginx is public and all health/startup boundaries behave as specified.
- [ ] No secret, raw data, backup, TLS key, or duplicate font/license asset is tracked.
- [ ] Tasks 1–5 pass from clean nested checkouts.
- [ ] Only Nginx is public; migration/import/admin/font/SSE boundaries behave correctly.
- [ ] No secret/raw data/runtime/font duplication enters deployment Git.
## Task 6: Implement release preflight, deploy, and rollback
## Task 6: Adapt Qlockify manual deploy and rollback
**Description:** Build the serialized manifest-driven operator workflow for preflight, migration, service replacement, postflight, observation, active/previous switch, and rollback.
**Description:** Modify Qlockify's deploy flow for ZarinPal paths/services and three-hash release records while removing unconditional destructive Git synchronization.
**Acceptance criteria:**
- [ ] Deploy uses image digests only and records evidence without secrets.
- [ ] Candidate health/migration/font failure preserves/restores prior release.
- [ ] Application-only rollback completes within 15 minutes and never auto-runs down migration.
- [ ] Latest `origin/main` is fast-forwarded and three hashes recorded before build.
- [ ] Dirty/diverged checkout, pull, build, migration, or health failure preserves the active release.
- [ ] Previous hashes rebuild/restore within 15 minutes without automatic down migration.
**Verification:** Shell/unit tests plus induced staging failures and timed rollback drill.
**Verification:** Script tests plus induced staging failures and timed rollback drill.
**Dependencies:** Checkpoint A
**Files likely touched:** deploy/release scripts and focused tests (maximum 5 files per incremental slice)
**Files likely touched:** deploy script/release helper/tests/docs, maximum 5 files per slice
**Estimated scope:** Split into multiple Medium tasks during implementation
**Estimated scope:** Medium
## Task 7: Implement deployment-repository CI/CD
## Task 7: Adapt Qlockify encrypted backup/upload
**Description:** Validate docs/config/Compose/Nginx/scripts/policies, accept authenticated candidates, enforce protected approval/locking, and invoke the manifest-driven release.
**Description:** Reuse Qlockify's PostgreSQL dump, OpenSSL, rclone upload, and retention flow with ZarinPal paths/services/credentials and a non-secret manifest.
**Acceptance criteria:**
- [ ] PR workflows validate only and cannot access production secrets/deploy.
- [ ] Production release requires gates plus human approval and is serialized.
- [ ] Provenance/SBOM/scan/secret results attach to the release record.
- [ ] Archive/listing/checksums/encryption/remote verification gate success.
- [ ] Exactly three verified encrypted local and seven remote backups are retained.
- [ ] `.env`, secrets, Redis, TLS, CSV, credentials, logs, monitoring, font, and license are excluded.
**Verification:** Forge event tests with denied PR deploy, concurrent candidate events, failed gate, and approved release.
**Verification:** Fixture database/object-store tests, exclusion inspection, failed-upload/cleanup tests.
**Dependencies:** Task 6 and confirmed forge/runner/registry versions
**Dependencies:** Tasks 1–3 and approved backup inputs
**Files likely touched:** workflow plus CI-focused helpers/tests (maximum 5 files)
**Files likely touched:** adapted backup/upload scripts/tests/docs, maximum 5 files per slice
**Estimated scope:** Medium (3–5 files)
**Estimated scope:** Medium
## Task 8: Coordinate backend/frontend candidate workflows
## Task 8: Adapt Qlockify guarded restore
**Description:** In separately approved service-repository changes, publish one immutable tested image and send candidate metadata without direct production SSH.
**Description:** Reuse Qlockify download/decrypt/restore flow while requiring explicit backup id, exact destination, empty target, and all production guards.
**Acceptance criteria:**
- [ ] Backend gates include Go/migration/import/Asynq queue/recovery verification.
- [ ] Frontend gates include Svelte/accessibility/RTL/build/font verification.
- [ ] Service events cannot mutate production and use least-privilege credentials.
- [ ] Every absent/mismatched production factor aborts before mutation.
- [ ] No environment/secret/Redis restoration occurs.
- [ ] Clean restore passes schema/count/tenant/login/cold-cache/worker/SSE checks within two hours.
**Verification:** Each repository's CI on PR/main plus deployment candidate receipt test.
**Verification:** Disposable-database tests, guard negative matrix, timed rehearsal.
**Dependencies:** Task 7 and separate authorization in each service repository
**Dependencies:** Task 7
**Files likely touched:** owned service workflow/tests only; not this deployment task
**Files likely touched:** adapted restore scripts/tests/docs, maximum 5 files per slice
**Estimated scope:** Separate cross-repository tasks
**Estimated scope:** Medium
## Task 9: Implement observability baseline
## Task 9: Bootstrap and accept MVP server
**Description:** Add bounded structured logging, private metrics/exporters/dashboards, release correlation, and symptom alerts/runbook links.
**Description:** Follow the reviewed bootstrap runbook, deploy manually, run one-time data/demo setup, and capture MVP evidence.
**Acceptance criteria:**
- [ ] All required RED/USE/worker/SSE/backup/TLS signals exist with bounded labels.
- [ ] Logs rotate and contain request/release correlation but no forbidden fields.
- [ ] Page/ticket alerts are test-fired and monitoring is not public.
- [ ] Access/firewall/Docker/DNS/TLS/deployment boundaries pass.
- [ ] Three-hash release, migration, import, accounts, tenant isolation, worker/SSE, and font pass.
- [ ] MVP manual health/log/disk/certificate/queue checks and encrypted backup/restore pass.
**Verification:** Induced staging failure located from telemetry, cardinality/redaction checks, alert delivery test.
**Verification:** Completed bootstrap/acceptance record.
**Dependencies:** Checkpoint A and approved monitoring versions/budgets
**Dependencies:** Tasks 6–8 and approved server/window
**Files likely touched:** Compose monitoring slice/config/dashboards/tests, split to <=5 files per commit
**Estimated scope:** Split into multiple Medium tasks
## Task 10: Implement encrypted backup and retention
**Description:** Produce verified logical PostgreSQL backups, encrypt before durable storage, upload/verify remotely, retain three local/seven remote, and emit metrics.
**Acceptance criteria:**
- [ ] Archive/listing/checksums/manifest/remote verification gate success.
- [ ] Backup contains none of the explicit exclusions.
- [ ] Cleanup never removes the last known-good copy and uses exact validated paths.
**Verification:** Backup tests against fixture DB/object store, archive exclusion inspection, failed-upload/cleanup tests.
**Dependencies:** Tasks 1–3 and approved encryption/provider decisions
**Files likely touched:** backup script/config/tests/docs (maximum 5 files per slice)
**Estimated scope:** Medium (3–5 files)
## Task 11: Implement guarded restore and rehearsal
**Description:** Validate/decrypt/trust-check an explicit backup, require all production guards, restore into an empty destination, and run application validation.
**Acceptance criteria:**
- [ ] Each absent/mismatched production factor aborts before mutation.
- [ ] Partial/failed restore is never treated as usable or blindly retried.
- [ ] Clean rehearsal passes tenant/session/cold-cache/worker/SSE checks within two hours.
**Verification:** Destructive tests only against exact disposable fixtures, production-guard negative matrix, timed rehearsal.
**Dependencies:** Task 10 and approved switch strategy
**Files likely touched:** restore script/tests/docs (maximum 5 files per slice)
**Estimated scope:** Medium (3–5 files)
## Checkpoint B: Release and recovery
- [ ] Tasks 6–11 pass in staging with release/rollback/backup/restore evidence.
- [ ] Registry, runner, monitoring, object storage, and secrets remain least privilege/private.
## Task 12: Bootstrap and accept first production-shaped server
**Description:** Follow the reviewed bootstrap runbook, deploy an approved manifest, run the one-time data/demo setup, and capture every acceptance criterion.
**Acceptance criteria:**
- [ ] Access/firewall/Docker/DNS/TLS/deploy/monitoring boundaries pass.
- [ ] Migration/import/demo accounts/tenant isolation/analytics SSE pass.
- [ ] Encrypted remote backup and clean restore rehearsal meet RPO/RTO.
**Verification:** Completed runbook evidence and stakeholder acceptance review.
**Dependencies:** Checkpoint B and approved host/change window
**Files likely touched:** evidence/release records only; no secret/raw runtime artifacts
**Files likely touched:** non-sensitive release/evidence records only
**Estimated scope:** Operational session
## Final checkpoint
## Checkpoint B: MVP ready
- [ ] Every measurable acceptance criterion in the approved spec passes.
- [ ] Active and previous releases are exact and rollback-ready.
- [ ] Remaining decisions are resolved or explicitly defer launch.
- [ ] Documentation reflects implemented reality and no implementation gate was bypassed.
- [ ] Every MVP acceptance criterion passes.
- [ ] Active/previous three-hash releases and rollback are ready.
- [ ] Clean restore meets RPO/RTO.
- [ ] Stakeholder explicitly confirms the core project is ready, opening the observability phase.
## Task 10: Add Prometheus and private exporters
**Description:** Add post-MVP collection for application, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS signals.
**Acceptance criteria:**
- [ ] Required signals appear with bounded labels and retention.
- [ ] Prometheus/exporters are private and resource-limited.
- [ ] Merchant/user/request/card identifiers are absent from metric labels.
**Verification:** Target health, signal/cardinality/resource tests.
**Dependencies:** Checkpoint B
**Files likely touched:** split into <=5-file monitoring slices
**Estimated scope:** Multiple Medium slices
## Task 11: Add Grafana dashboards and alerts
**Description:** Add private dashboards, release correlation, actionable symptom alerts, runbook links, and delivery testing.
**Acceptance criteria:**
- [ ] Service, worker/SSE, data, host, backup, and TLS dashboards answer the spec's operator questions.
- [ ] Only page/ticket alerts exist and each is actionable.
- [ ] Induced failures alert and can be diagnosed without source archaeology.
**Verification:** Dashboard review, induced failure, alert test-fire, redaction check.
**Dependencies:** Task 10
**Files likely touched:** split into <=5-file dashboard/alert slices
**Estimated scope:** Multiple Medium slices
## Final observability checkpoint before AI
- [ ] Prometheus/Grafana phase meets its acceptance criteria.
- [ ] Alert delivery and runbook ownership are proven.
- [ ] Stakeholder explicitly approves beginning AI implementation.
- [ ] No CI/CD pipeline or automated deployment system was introduced.