diff --git a/tasks/plan.md b/tasks/plan.md index 64a6208..819f5db 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -1,126 +1,146 @@ # Implementation Plan: Platform Operations -- Status: Implementation plan ready for stakeholder review — implementation not authorized +- Status: Ready for stakeholder review; implementation not authorized - Capability: Platform operations - Task list target: `tasks/todo.md` +- Planning baseline: `docs/QLOCKIFY-ADAPTATION-CONTRACT.md` - Last updated: 2026-08-20 -## Confirmed implementation direction +## Outcome -Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds. +Adapt the read-only Qlockify deployment repository into a manual, production-shaped ZarinPal Analytics deployment for `zarinpal.amiirkhl.ir`. Planning does not wait for a server IP, live DNS, real credentials, or named operators. Tracked artifacts use safe defaults and blank secret placeholders; real values are supplied only when an operation needs them. -There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work. +There is no CI/CD pipeline, runner, registry, automated Git synchronization, or private font injection. Prometheus/Grafana remain a required post-MVP, pre-AI phase. -No task below is authorized merely because it appears here. +No task below authorizes implementation. ## Dependency graph ```text -resolve concrete server/runtime inputs + authorize build - | - +--> safe config + manual release-record contract - | | - | +--> Compose data/app services - | | | - | | +--> migrate/import/admin separation - | | +--> manual deploy/rollback - | | - | +--> Nginx/Certbot routing and TLS - | - +--> Qlockify backup adaptation --> guarded restore --> rehearsal - | - +--> first-server bootstrap --> MVP acceptance - | - v - Prometheus/Grafana phase - | - v - AI may begin +approved spec + Qlockify adaptation contract + explicit build authorization + | + +--> environment examples/validation --> data-service Compose + | | | + | +--> release record/lock +--> API/worker/one-offs + | | + +--> frontend build --------------------------------+ + | + Nginx routing --> Certbot/TLS + | + first-start --> deploy --> rollback + | | + +-----------+--> backup/upload + | + guarded restore + | + first-server MVP gate + | + Prometheus --> Grafana + | + AI may begin ``` -## Architecture decisions +## Fixed architecture decisions -- Preserve ADR 0001's three independent repositories and ignored nested service checkouts. -- Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records. -- Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license. -- Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags. -- Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup. -- Separate API, worker, migration, one-time CSV import, and administrative processes. -- Keep Asynq pinned to pre-1.0 `v0.26.0`. -- Use automatic Let's Encrypt/Certbot TLS. -- Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI. +- Preserve ADR 0001's three independent repositories and direct ignored `backend/` and `frontend/` checkouts. +- Follow ADR 0002: the operator pulls all repositories interactively before deployment; scripts verify/record but do not pull or reset Git. +- Follow ADR 0003/frontend commit `64f5ac2`: consume the tracked font/license through the normal frontend build context. +- Pin `postgres:18.4-alpine3.24`, `redis:7.4.10-alpine3.21`, `nginx:1.29.8-alpine3.23`, and `certbot/certbot:v5.7.0`. +- Keep PostgreSQL authoritative. Redis uses AOF `everysec`/`noeviction` but remains disposable and excluded from backup. +- Keep API, worker, migrate, importer, and admin commands separate; Asynq stays explicitly pre-1.0 at `v0.26.0`. +- Reuse Qlockify's root/backend/frontend `.env` shape and backup/restore script family with the differences recorded in the adaptation contract. +- Use structured logs and manual operator checks for MVP; add private Prometheus/Grafana only after MVP acceptance and before AI. -## Phase 0: resolve implementation inputs and authorize the build +## Phase 0: planning baseline -- Treat the approved Platform Operations spec and ADRs 0001–0003 as the implementation baseline. -- Obtain the production IP/DNS state, repository URLs, actual protected environment values, ParsPack endpoint/bucket/prefix/credentials, and named operator/secret-escrow contacts. -- Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery. -- Obtain the frontend build/static health contract. -- Obtain explicit authorization before creating deployment implementation artifacts. +- The Platform Operations spec, ADRs 0001–0003, Qlockify adaptation contract, bootstrap runbook, and backup/restore runbook define the approved behavior. +- Runtime values such as IP, DNS access, email, ParsPack credentials, and encryption passphrase are deliberately deferred to the operation that uses them. +- Backend/frontend command and variable names are integrated when their implementation contracts exist; earlier independent deployment work is not blocked. +- Implementation starts only after a separate explicit stakeholder authorization. -Checkpoint: no implementation file exists before explicit build authorization and the inputs needed by its slice are resolved. +Checkpoint: the plan and tasks are approved; no implementation artifact exists yet. -## Phase 1: configuration and Compose foundation +## Phase 1: configuration and runtime foundation -- Add safe tracked configuration examples and validate protected external secret paths. -- Define non-secret active/previous release records containing the three SHAs and health/migration evidence. -- Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles. -- Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts. +1. Create the deployment-root example and redaction-safe validation for the Qlockify-shaped three-file layout; consume backend/frontend-owned examples without duplicating them. +2. Implement private PostgreSQL/Redis services, named volumes, health checks, authentication, resource limits, and bounded logs. +3. Add API/worker plus separate migration/import/admin profiles from one backend revision. +4. Add the ordinary frontend static build and font/static health checks. -Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin. +Checkpoint: Compose renders from placeholder test values; only intended networks exist; ordinary startup cannot invoke migration/import/admin; no secret/font/runtime artifact is tracked. -## Phase 2: Nginx, TLS, and lifecycle +## Phase 2: edge and first-start lifecycle -- Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`. -- Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior. -- Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset. -- Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release. +5. Adapt Qlockify Nginx routing for the single canonical domain, API, frontend, readiness, and exact SSE behavior. +6. Adapt Qlockify Certbot issuance/renewal/reload for the single domain and pinned image. +7. Add guarded first-start orchestration for data readiness, migration, one-time import/reconciliation, and first public activation. -Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes. +Checkpoint: routing/TLS configuration validates, first-start fails closed before public activation, and later ordinary startup never imports. -## Phase 3: backup, restore, and first-server acceptance +## Phase 3: releases and rollback -- Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials. -- Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only. -- Add exact destination and production confirmation guards. -- Bootstrap the server with `first-start.sh`: migrate, import/reconcile the pinned archive, then activate public services; perform separate credential setup, encrypted remote backup, and a clean restore rehearsal within two hours. +8. Define the ignored release-record format and exclusive operation lock. +9. Adapt Qlockify's deploy flow to verify manually pulled clean revisions, build/migrate/replace services, observe health, and activate one three-hash release. +10. Implement prior-release checkout/rebuild rollback with migration compatibility guards. -Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks. +Checkpoint: induced checkout/build/migration/readiness failures preserve the active stack; a prior application release is recoverable within 15 minutes under normal conditions. -## Phase 4: required observability before AI +## Phase 4: backup and restore -- Add Prometheus, Grafana, and minimum private exporters/instrumentation. -- Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards. -- Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery. -- Validate the observability acceptance gate before any AI implementation starts. +11. Adapt Qlockify's plain-SQL PostgreSQL backup to emit only database data plus a non-secret manifest/checksums. +12. Adapt its OpenSSL/rclone upload and encrypted 3-local/7-remote retention flow for the ZarinPal ParsPack prefix. +13. Harden local restore with explicit backup, clean-target, compatibility, and production confirmation guards. +14. Adapt S3 download/decrypt to verify an explicit remote object and hand off to the guarded restore. -Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator. +Checkpoint: excluded content inspection passes, failed upload/retention cannot delete the last good copy, and a clean rehearsal meets RPO 24 hours/RTO two hours. + +## Phase 5: first-server MVP acceptance + +15. Follow the bootstrap runbook on the prepared Ubuntu server, fill real environment values, deploy/import once, verify the application, rehearse rollback/restore, and record evidence. + +Checkpoint: all MVP criteria pass. The stakeholder explicitly marks MVP ready before monitoring work begins. + +## Phase 6: required observability before AI + +16. Add private Prometheus and approved exporters/instrumentation with bounded labels, retention, and resource use. +17. Add private Grafana dashboards, release correlation, symptom-based alerts, runbook links, and tested delivery. + +Checkpoint: an induced failure is detected and diagnosed from telemetry; the stakeholder approves the observability gate before AI begins. + +## Verification strategy + +- Documentation/planning: Markdown diff checks, source mapping review, secret-pattern scan, and no implementation files. +- Configuration: placeholder-only rendering, missing/invalid/unknown variable tests, permission/exclusion checks, and no resolved-value output. +- Compose: configuration validation, service health, network/port inspection, resource/log limits, profiles, and restart/Redis-loss behavior. +- Shell operations: ShellCheck plus disposable fixture tests for exact paths, locks, dirty/diverged Git, failure cleanup, retention, and restore guard matrices. +- Edge: `nginx -t`, canonical/unknown-host behavior, security headers, TLS issuance/renewal, static/font behavior, API routing, and streamed SSE checks. +- Release/recovery: induced deployment failures, timed prior-hash rollback, backup inspection, remote verification, and clean-database restore rehearsal. +- MVP/observability: manual acceptance record first; monitoring target/cardinality/dashboard/alert tests only after the MVP gate. ## Risks and mitigations | Risk | Impact | Mitigation | |---|---|---| -| Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record | -| Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` | -| Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time | -| Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration | -| Redis loss drops queue/cache | Partial analytics interruption | AOF `everysec`, PostgreSQL authority, idempotent reconciliation/cold-cache tests | -| SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test | -| Docker-group compromise | Host-root compromise | Restricted key-only access to trusted `ubuntu` account, no daemon TCP exposure | -| Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests | -| Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal | -| Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI | +| Qlockify-specific behavior is copied blindly | Wrong services or unsafe operations | File-by-file adaptation matrix and explicit retain/change/drop review | +| Environment ownership drifts | Secret mismatch or accidental client exposure | Root/backend/frontend ownership contract; one owner per variable; redaction-safe validation | +| Manual releases select inconsistent source | Broken stack | Interactive pulls first, one lock, three clean SHA checks, one release record | +| Server-side builds vary over time | Difficult reproduction | Pin infrastructure/dependency inputs and record three SHAs plus local image ids | +| Migration prevents rollback | Extended outage | Expand/contract sequencing, preflight backup, no automatic down migration | +| Backup contains secrets or is unusable | Disclosure or data loss | PostgreSQL-only package, encryption/checksums, explicit exclusions, clean restore rehearsal | +| Redis loss interrupts analytics | Temporary missing results | AOF `everysec`, PostgreSQL authority, idempotent reconciliation and cold-cache tests | +| 2-GB server exhausts memory | OOM/restarts | Approved per-service limits, 2-GB swap, bounded imports/logs, capacity upgrade when measured | +| Monitoring is postponed too far | Weak diagnosis before AI | Hard MVP-to-observability-to-AI phase gate | -## Parallelization after approval +## Parallel and sequential work -- Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed. -- Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate. -- Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced. +- After environment/paths are fixed, PostgreSQL/Redis foundation, frontend build, Nginx fixtures, and backup fixtures can be prepared in separate focused slices. +- Compose integration, migration/import, release activation, rollback, production restore, DNS/TLS cutover, and phase gates remain sequential. +- Backend/frontend repositories define their application commands and variables; deployment consumes those contracts without editing those repositories in this workstream. ## Completion - Every authorized task in `tasks/todo.md` passes its acceptance and verification. -- The active stack maps to exact deployment/backend/frontend commits. -- Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass. -- Prometheus/Grafana acceptance passes before AI begins. -- Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets. +- The active stack maps to exact deployment/backend/frontend commits and can roll back safely. +- Only Nginx publishes application ports; secrets/runtime/raw/font copies remain outside deployment Git. +- Encrypted off-host backup and clean restore meet their RPO/RTO. +- Prometheus/Grafana acceptance passes after MVP and before AI. diff --git a/tasks/todo.md b/tasks/todo.md index f0cf6a1..61b434c 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -1,230 +1,348 @@ # Platform Operations implementation tasks -- Status: Ready for stakeholder review — implementation not authorized +- Status: Planned and ready for stakeholder review; implementation not authorized - Canonical plan: `tasks/plan.md` +- Adaptation baseline: `docs/QLOCKIFY-ADAPTATION-CONTRACT.md` -## Task 0: Resolve implementation inputs and preserve the implementation gate +## Task 0: Approve the implementation plan -**Description:** Use the approved manual Qlockify-style specification to collect the concrete values and service contracts required for later implementation while preserving the separate explicit build-authorization gate. +**Description:** Confirm that the Qlockify adaptation contract, dependency order, task boundaries, and verification gates are the correct basis for later implementation. **Acceptance criteria:** -- [x] Revised Platform Operations spec is explicitly approved. -- [x] ADRs 0001–0003 match the approved topology, manual release, and font decisions. -- [ ] Production connection values, protected secrets, repository URLs, service commands/routes, and responsible operators are recorded outside public Git where required. -- [ ] Stakeholder explicitly authorizes creation of deployment implementation artifacts. +- [x] Platform Operations specification and ADRs 0001–0003 are approved. +- [x] Qlockify flow/environment mapping is documented without real runtime values. +- [ ] Stakeholder approves this plan/task breakdown and separately authorizes implementation when ready. -**Verification:** Approved specification, clean documentation review, resolved slice inputs, and explicit build authorization. +**Verification:** Documentation diff review and confirmation that no implementation artifact was created. **Dependencies:** None -**Files likely touched:** spec and ADRs only +**Files likely touched:** documentation only **Estimated scope:** Small -## Task 1: Define safe configuration and release records +## Task 1: Add the deployment environment contract -**Description:** Define tracked placeholders, external protected secrets, deployment locking, and non-secret active/previous release records. +**Description:** Add the root deployment example and redaction-safe validation for the Qlockify-shaped root/backend/frontend environment layout. Backend and frontend examples remain owned by their repositories and are consumed without duplication. **Acceptance criteria:** -- [ ] Release record contains three SHAs, local image ids, migration, backup, operator, and health evidence. -- [ ] Configuration examples contain no real or plausible secret values. -- [ ] Font provenance uses frontend commit/build only; no second asset path exists. +- [ ] The root example contains only deployment-owned safe defaults/blank placeholders; every service-owned variable has one documented owner. +- [ ] Real `.env` files stay ignored, mode `0600`, excluded from backup, and absent from rendered/logged output. +- [ ] Validation rejects missing, unknown, malformed, or client-exposed secret variables for the selected operation. -**Verification:** Schema/format tests, secret scan, invalid-record tests. +**Verification:** Placeholder render, negative validation matrix, Git-ignore/permission checks, and staged secret scan. -**Dependencies:** Task 0 +**Dependencies:** Task 0 plus explicit implementation authorization -**Files likely touched:** examples/schema/tests/docs, maximum 5 files +**Files likely touched:** `.env.example`, validation helper/test, documentation; maximum 5 files **Estimated scope:** Medium -## Task 2: Implement PostgreSQL and Redis Compose foundation +## Task 2: Add PostgreSQL and Redis Compose foundation -**Description:** Add pinned PostgreSQL 18 and Redis 7 Alpine, private data network, volumes, authentication, health, resources, and approved Redis AOF choice. +**Description:** Define pinned private data services, health checks, named volumes, authentication, resource limits, bounded logs, and Redis AOF behavior. **Acceptance criteria:** -- [ ] Both services become healthy with explicit limits/tags. -- [ ] Neither publishes a host port. -- [ ] Redis is excluded from backup and loss does not affect PostgreSQL authority. +- [ ] PostgreSQL and Redis become healthy using the approved pins and memory limits. +- [ ] Neither publishes a host port; only approved services can join the private data network. +- [ ] Redis uses AOF `everysec`, `noeviction`, 192-MB maxmemory, and remains excluded from backup. -**Verification:** Compose validation, health, port/network, volume, Redis-loss tests. +**Verification:** Compose validation, health/port/network/volume/resource inspection, restart and Redis-loss tests. **Dependencies:** Task 1 -**Files likely touched:** Compose/config/tests, maximum 5 files +**Files likely touched:** `compose.yaml`, data config, focused validation/test files; maximum 5 files **Estimated scope:** Medium -## Task 3: Implement backend, worker, and one-off boundaries +## Task 3: Add backend runtime boundaries -**Description:** Wire API/worker plus separate migration/import/admin commands and graceful lifecycle. +**Description:** Wire API and worker services plus one-off migrate/import/admin profiles from the same recorded backend checkout/image lineage. **Acceptance criteria:** -- [ ] Migration completes before API/worker readiness. -- [ ] Normal startup never runs import/admin; import uses one exact read-only path. -- [ ] `first-start.sh` starts data services, migrates, imports and reconciles the pinned archive, then starts public services only after success. -- [ ] Worker is private/event-driven and uses pinned pre-1.0 Asynq `v0.26.0`. +- [ ] API and worker use private networks, health/readiness, graceful shutdown, and approved resource limits. +- [ ] Migrate/import/admin are explicit one-off profiles and never run during ordinary startup. +- [ ] Worker uses pre-1.0 Asynq `v0.26.0` with concurrency 16 globally and 4 per merchant. -**Verification:** First-start, ordinary-startup, migration/import failure, checksum/count reconciliation, idempotent no-op, and shutdown integration tests. +**Verification:** Compose profile/startup/failure/shutdown tests and service command-contract review. -**Dependencies:** Task 2 and approved backend commands +**Dependencies:** Task 2 and the backend-owned command contract -**Files likely touched:** Compose/service wrappers/tests, maximum 5 files per slice +**Files likely touched:** `compose.yaml`, backend image definition, lifecycle tests/docs; maximum 5 files **Estimated scope:** Medium -## Task 4: Implement frontend build and font health +## Task 4: Add frontend production build -**Description:** Build the nested frontend checkout normally and verify static SPA/font output without deployment copies or injection. +**Description:** Build and serve the direct frontend checkout using its ordinary static context and same-origin API configuration. **Acceptance criteria:** -- [ ] Build uses recorded frontend commit at/after `64f5ac2`. -- [ ] Expected font resource returns 200 with correct content type. -- [ ] Deployment repository contains no font binary, `FontLicense.txt`, fetch, credential, or alternate path. +- [ ] Build consumes frontend commit `64f5ac2` or later and its tracked font/license normally. +- [ ] Static sentinel, SPA fallback, and emitted font resource/content type pass. +- [ ] Deployment Git contains no font/license copy, fetch, credentials, or alternate asset path. -**Verification:** Build/static health plus repository-policy scan. +**Verification:** Production build/static-server checks plus repository-policy scan. -**Dependencies:** Task 1 and approved frontend build contract +**Dependencies:** Task 1 and the frontend-owned build contract -**Files likely touched:** frontend deployment wrapper/service/tests, maximum 5 files +**Files likely touched:** frontend image definition, `compose.yaml`, focused health/tests; maximum 5 files **Estimated scope:** Medium -## Task 5: Implement Nginx and automatic TLS +## Checkpoint A: Runtime foundation -**Description:** Adapt Qlockify's Nginx/Certbot pattern for canonical routing, Let's Encrypt, API/frontend separation, exact SSE behavior, security headers, and health. +- [ ] Tasks 1–4 pass from clean nested checkouts with placeholder test values. +- [ ] Normal startup cannot invoke migration/import/admin. +- [ ] No secrets, runtime state, raw data, or duplicated font assets are tracked. + +## Task 5: Adapt Qlockify Nginx routing + +**Description:** Adapt the Qlockify template/entrypoint for the single canonical domain, default-host rejection, frontend/API routing, security headers, health, caching, and SSE. **Acceptance criteria:** -- [ ] Only 80/443 are public; unknown hosts reject and canonical redirect is exact. -- [ ] Staging/production issuance and renewal/reload pass. -- [ ] SSE heartbeat/reconnect streams without buffering/cache/compression. +- [ ] Only ports 80/443 publish; HTTP redirects exactly to `zarinpal.amiirkhl.ir`; unknown hosts reject. +- [ ] API and frontend routing/fallback do not overlap and public readiness remains generic. +- [ ] Exact SSE route preserves auth/event ids and streams heartbeats without buffering, cache, compression, or premature timeout. -**Verification:** `nginx -t`, external TLS/header/routing checks, streamed SSE test. +**Verification:** `nginx -t`, route/header/default-host tests, static/font test, and streamed SSE fixture. -**Dependencies:** Tasks 3–4 and approved routes/CSP +**Dependencies:** Tasks 3–4 -**Files likely touched:** Nginx/Certbot assets/tests, maximum 5 files per slice +**Files likely touched:** Nginx template, image/entrypoint, focused tests; maximum 5 files **Estimated scope:** Medium -## Checkpoint A: Runtime MVP foundation +## Task 6: Adapt Qlockify Certbot lifecycle -- [ ] Tasks 1–5 pass from clean nested checkouts. -- [ ] Only Nginx is public; migration/import/admin/font/SSE boundaries behave correctly. -- [ ] No secret/raw data/runtime/font duplication enters deployment Git. - -## Task 6: Adapt Qlockify manual deploy and rollback - -**Description:** Modify Qlockify's deploy flow for ZarinPal paths/services and three-hash release records while keeping interactive Git pulls as a separate operator prerequisite and removing unconditional destructive Git synchronization. +**Description:** Adapt Qlockify's HTTP-01 bootstrap, pinned Certbot service, certificate watcher, renewal, and Nginx reload for the single domain. **Acceptance criteria:** -- [ ] Operator manually fast-forwards all three repositories to latest `origin/main`; the deployment command performs no pull and records all three hashes before build. -- [ ] Dirty/diverged checkout, build, migration, or health failure preserves the active release; a manual pull failure prevents deployment from starting. -- [ ] Previous hashes rebuild/restore within 15 minutes without automatic down migration. +- [ ] Missing production certificate uses only the bounded bootstrap certificate path until issuance. +- [ ] Staging issuance, production issuance, renewal, and validated Nginx reload pass. +- [ ] No alternate domain, local production certificate mode, private key, or ACME state enters Git. -**Verification:** Script tests plus induced staging failures and timed rollback drill. +**Verification:** Compose/entrypoint tests, staging ACME drill, certificate name/expiry/protocol inspection. -**Dependencies:** Checkpoint A +**Dependencies:** Task 5 -**Files likely touched:** deploy script/release helper/tests/docs, maximum 5 files per slice +**Files likely touched:** Certbot entrypoint/config, `compose.yaml`, focused tests/docs; maximum 5 files **Estimated scope:** Medium -## Task 7: Adapt Qlockify encrypted backup/upload +## Task 7: Add guarded first-start orchestration -**Description:** Reuse Qlockify's PostgreSQL dump, OpenSSL, rclone upload, and retention flow with ZarinPal paths/services/credentials and a non-secret manifest. +**Description:** Add the explicit first-server path that starts data services, migrates, imports/reconciles the pinned CSV, and activates public services only after success. **Acceptance criteria:** -- [ ] Package contents, plain-SQL header/checksums, encryption, and remote verification gate success. -- [ ] Exactly three verified encrypted local and seven remote backups are retained. -- [ ] `.env`, secrets, Redis, TLS, CSV, credentials, logs, monitoring, font, and license are excluded. +- [ ] Input is exact/read-only and migration/import/reconciliation failures prevent public activation. +- [ ] A completed identical import is a safe no-op; recovery never guesses a partial CSV position. +- [ ] Ordinary deploy/start/reboot paths never invoke first-start, importer, or admin jobs. -**Verification:** Fixture database/object-store tests, exclusion inspection, failed-upload/cleanup tests. +**Verification:** Success, checksum/schema/count, interruption/recovery, idempotency, and ordinary-start negative tests. -**Dependencies:** Tasks 1–3 and approved backup inputs +**Dependencies:** Tasks 2–6 and approved backend importer commands -**Files likely touched:** adapted backup/upload scripts/tests/docs, maximum 5 files per slice +**Files likely touched:** `scripts/first-start.sh`, Compose profile, focused tests/docs; maximum 5 files **Estimated scope:** Medium -## Task 8: Adapt Qlockify guarded restore +## Checkpoint B: Edge and first-start -**Description:** Reuse Qlockify download/decrypt/restore flow while requiring explicit backup id, exact destination, empty target, and all production guards. +- [ ] Nginx/TLS/static/API/SSE checks pass. +- [ ] First-start is fail-closed and ordinary startup never imports. +- [ ] Only Nginx is externally reachable. + +## Task 8: Add release records and operation lock + +**Description:** Define ignored current/previous/timestamped release records and one exclusive lock shared by deploy, rollback, restore, migration, and import as required. **Acceptance criteria:** -- [ ] Every absent/mismatched production factor aborts before mutation. -- [ ] No environment/secret/Redis restoration occurs. -- [ ] Clean restore passes schema/count/tenant/login/cold-cache/worker/SSE checks within two hours. +- [ ] Record captures three SHAs, local image ids, migration, backup, operator, health, outcome, and previous release without secrets. +- [ ] Invalid/incomplete records cannot become active and current/previous updates are atomic. +- [ ] Conflicting operations fail safely without stale-lock ambiguity. -**Verification:** Disposable-database tests, guard negative matrix, timed rehearsal. +**Verification:** Schema/parser tests, invalid-record matrix, lock contention/stale-owner tests, and redaction scan. -**Dependencies:** Task 7 +**Dependencies:** Task 1 -**Files likely touched:** adapted restore scripts/tests/docs, maximum 5 files per slice +**Files likely touched:** release schema/helper/tests/docs; maximum 5 files **Estimated scope:** Medium -## Task 9: Bootstrap and accept MVP server +## Task 9: Adapt Qlockify manual deployment -**Description:** Follow the reviewed bootstrap runbook, run the guarded first-start migration/import/application sequence, perform separate demo setup, and capture MVP evidence. +**Description:** Adapt Qlockify's deploy structure to verify three manually pulled repositories, build/migrate/replace services, observe health, and activate one coherent release. **Acceptance criteria:** -- [ ] Access/firewall/Docker/DNS/TLS/deployment boundaries pass. -- [ ] Three-hash release, migration, import, accounts, tenant isolation, worker/SSE, and font pass. -- [ ] MVP manual health/log/disk/certificate/queue checks and encrypted backup/restore pass. +- [ ] Script performs no Git fetch/pull/reset and rejects wrong path/remote/branch/SHA/dirty state. +- [ ] Build, migration, readiness, font, worker, or SSE failure leaves the active release unchanged and records evidence. +- [ ] Successful deployment records all three SHAs and passes a 15-minute postflight observation before activation. -**Verification:** Completed bootstrap/acceptance record. +**Verification:** ShellCheck, disposable Git fixtures, Compose mocks/integration, and induced failure matrix. -**Dependencies:** Tasks 6–8 and approved server/window +**Dependencies:** Tasks 5–8 -**Files likely touched:** non-sensitive release/evidence records only +**Files likely touched:** `scripts/deploy.sh`, release helper, focused tests/runbook; maximum 5 files + +**Estimated scope:** Medium + +## Task 10: Add three-hash rollback + +**Description:** Select the previous release record, verify its three revisions, rebuild, and run the normal postflight without automatic database down-migration. + +**Acceptance criteria:** +- [ ] Exact previous hashes and clean repository guards precede checkout/build. +- [ ] Migration incompatibility stops rollback with the current stack preserved. +- [ ] Normal application rollback completes within 15 minutes and is recorded as a new event. + +**Verification:** Git fixture tests, incompatible-migration negative test, and timed rollback drill. + +**Dependencies:** Task 9 + +**Files likely touched:** rollback script/helper/tests/runbook; maximum 5 files + +**Estimated scope:** Medium + +## Checkpoint C: Release safety + +- [ ] Dirty/diverged/manual-pull/build/migration/readiness failures preserve the active stack. +- [ ] Active and previous releases map to exact three-repository revisions. +- [ ] Timed rollback meets its target without a down migration. + +## Task 11: Adapt Qlockify PostgreSQL backup + +**Description:** Retain Qlockify's matching-client plain-SQL dump flow while packaging only PostgreSQL plus a non-secret release/migration/checksum manifest. + +**Acceptance criteria:** +- [ ] Dump nonzero exit, empty output, invalid header, or checksum failure prevents package success. +- [ ] Package contains only plain SQL and the approved non-secret manifest/checksums. +- [ ] `.env`, secrets, Redis, TLS, CSV, credentials, logs, monitoring, font/license, and media are absent. + +**Verification:** Fixture database dump, package listing/header/checksum checks, and forbidden-content scan. + +**Dependencies:** Tasks 2, 8 + +**Files likely touched:** `scripts/backup.sh`, manifest helper/tests/runbook; maximum 5 files + +**Estimated scope:** Medium + +## Task 12: Adapt encrypted ParsPack upload and retention + +**Description:** Retain Qlockify's OpenSSL AES-256-CBC PBKDF2-200k and rclone structure while keeping three encrypted local and seven encrypted remote verified backups. + +**Acceptance criteria:** +- [ ] Upload uses a private temporary key and is finalized only after remote verification. +- [ ] Failed encryption/upload/verification preserves all previous copies and no plaintext backup remains durably. +- [ ] Retention removes only exact stale targets after new-copy verification and never deletes the last good copy. + +**Verification:** Fake S3/ParsPack-compatible fixture, failure/retention matrix, checksum and plaintext-residue checks. + +**Dependencies:** Task 11 + +**Files likely touched:** upload script, retention helper/tests/runbook; maximum 5 files + +**Estimated scope:** Medium + +## Task 13: Add guarded local restore + +**Description:** Restore an explicitly selected verified backup into a new empty target using `psql -X --set ON_ERROR_STOP=on` and independent production guards. + +**Acceptance criteria:** +- [ ] Provenance/checksum/header/version/free-space/exact-empty-target checks precede mutation. +- [ ] Every missing production factor aborts before writers stop or a database is changed. +- [ ] Restore never changes `.env`, Redis, TLS, CSV, credentials, logs, or frontend assets. + +**Verification:** Disposable database restore, negative guard matrix, partial-SQL failure, and exclusion checks. + +**Dependencies:** Task 11 + +**Files likely touched:** `scripts/restore.sh`, guard/helper/tests/runbook; maximum 5 files + +**Estimated scope:** Medium + +## Task 14: Adapt encrypted S3 restore handoff + +**Description:** Download an explicit encrypted object, verify/decrypt it in a restrictive temporary directory, and hand it to the guarded local restore. + +**Acceptance criteria:** +- [ ] Production requires an explicit object id; an implicit `latest` cannot select a production backup. +- [ ] Remote identity/size/checksum and decrypted package checks pass before restore invocation. +- [ ] Temporary encrypted/plaintext material is removed only from the exact validated temporary directory. + +**Verification:** Fake S3 object-selection/tamper tests, decryption failure tests, and guarded handoff rehearsal. + +**Dependencies:** Tasks 12–13 + +**Files likely touched:** `scripts/restore-from-s3.sh`, focused tests/runbook; maximum 4 files + +**Estimated scope:** Medium + +## Checkpoint D: Recovery + +- [ ] Daily schedule/3-local/7-remote retention behaves safely. +- [ ] Forbidden content is absent from every backup package. +- [ ] Clean restore passes schema/count/tenant/login/cold-cache/worker/SSE validation within two hours. + +## Task 15: Bootstrap and accept the MVP server + +**Description:** Execute the reviewed runbooks on the prepared Ubuntu server, fill runtime values, perform first-start, activate the release, and rehearse rollback/restore. + +**Acceptance criteria:** +- [ ] Host access/firewall/swap/Docker/DNS/TLS and only-80/443 exposure checks pass. +- [ ] Migration/import/accounts/tenant isolation/worker/SSE/frontend font and manual health/log checks pass. +- [ ] Three-hash release, rollback, encrypted remote backup, and clean restore meet their time/recovery targets. + +**Verification:** Signed non-secret bootstrap/MVP acceptance record. + +**Dependencies:** Checkpoints A–D and a prepared server/runtime values + +**Files likely touched:** ignored server evidence/release records only **Estimated scope:** Operational session -## Checkpoint B: MVP ready +## Checkpoint E: MVP ready - [ ] Every MVP acceptance criterion passes. -- [ ] Active/previous three-hash releases and rollback are ready. -- [ ] Clean restore meets RPO/RTO. -- [ ] Stakeholder explicitly confirms the core project is ready, opening the observability phase. +- [ ] Active/previous releases and backup/restore are proven. +- [ ] Stakeholder explicitly opens the observability phase. -## Task 10: Add Prometheus and private exporters +## Task 16: Add private Prometheus collection -**Description:** Add post-MVP collection for application, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS signals. +**Description:** Add post-MVP application, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS collection with bounded labels and retention. **Acceptance criteria:** -- [ ] Required signals appear with bounded labels and retention. -- [ ] Prometheus/exporters are private and resource-limited. +- [ ] Required signals and target health pass within approved resource/retention limits. +- [ ] Prometheus/exporters are private and not routed by Nginx. - [ ] Merchant/user/request/card identifiers are absent from metric labels. -**Verification:** Target health, signal/cardinality/resource tests. +**Verification:** Target/signal/cardinality/resource tests and sensitive-label scan. -**Dependencies:** Checkpoint B +**Dependencies:** Checkpoint E -**Files likely touched:** split into <=5-file monitoring slices +**Files likely touched:** split into focused Prometheus/exporter slices of no more than 5 files **Estimated scope:** Multiple Medium slices -## Task 11: Add Grafana dashboards and alerts +## Task 17: Add private Grafana dashboards and alerts -**Description:** Add private dashboards, release correlation, actionable symptom alerts, runbook links, and delivery testing. +**Description:** Add post-MVP dashboards, release correlation, actionable symptom alerts, runbook links, and tested operator delivery. **Acceptance criteria:** -- [ ] Service, worker/SSE, data, host, backup, and TLS dashboards answer the spec's operator questions. -- [ ] Only page/ticket alerts exist and each is actionable. -- [ ] Induced failures alert and can be diagnosed without source archaeology. +- [ ] Service, worker/SSE, data, host, backup, and TLS dashboards answer the approved operator questions. +- [ ] Every page/ticket alert is actionable, bounded, linked to a runbook, and tested through an induced failure. +- [ ] Grafana remains private and contains no sensitive labels, logs, or embedded credentials. -**Verification:** Dashboard review, induced failure, alert test-fire, redaction check. +**Verification:** Dashboard review, induced failure, alert test-fire, access and redaction checks. -**Dependencies:** Task 10 +**Dependencies:** Task 16 -**Files likely touched:** split into <=5-file dashboard/alert slices +**Files likely touched:** split into focused dashboard/alert slices of no more than 5 files **Estimated scope:** Multiple Medium slices ## Final observability checkpoint before AI -- [ ] Prometheus/Grafana phase meets its acceptance criteria. +- [ ] Prometheus/Grafana acceptance criteria pass after MVP. - [ ] Alert delivery and runbook ownership are proven. - [ ] Stakeholder explicitly approves beginning AI implementation. - [ ] No CI/CD pipeline or automated deployment system was introduced.