diff --git a/README.md b/README.md index 8d67526..9199cac 100644 --- a/README.md +++ b/README.md @@ -24,3 +24,9 @@ The intended production hostname is `zarinpal.amiirkhl.ir`. The deployment shape - Raw datasets and database state are never committed. - Every material change is a small Conventional Commit. +## Platform operations documents + +- `docs/PLATFORM-OPERATIONS-SPEC.md` defines the draft production topology, lifecycle, CI/CD, observability, security, backup/restore, acceptance, and operator boundaries. +- `docs/runbooks/FIRST-SERVER-BOOTSTRAP.md` defines the safe first-host sequence without implementing a bootstrap script. +- `docs/runbooks/BACKUP-RESTORE.md` defines logical PostgreSQL backup, encryption/retention, guarded restore, and rehearsal. +- `tasks/plan.md` and `tasks/todo.md` hold the approval-gated implementation plan. Their presence does not grant implementation authorization. diff --git a/tasks/plan.md b/tasks/plan.md new file mode 100644 index 0000000..851ecb6 --- /dev/null +++ b/tasks/plan.md @@ -0,0 +1,139 @@ +# Implementation Plan: Platform Operations + +- Status: Planned from draft specification; implementation blocked pending explicit approval +- Capability: Platform operations +- Task list target: `tasks/todo.md` +- Last updated: 2026-08-20 + +## Overview + +Implement the approved Platform Operations contract as a single-server, production-shaped Docker Compose platform for `zarinpal.amiirkhl.ir`. The deployment repository remains the only operational coordinator; backend and frontend source stay in their own repositories and publish immutable images. Work proceeds from decisions and validation contracts through topology, edge, lifecycle, release automation, recovery, and end-to-end proof. + +No task below is authorized merely because it appears in this plan. The Platform Operations spec must be approved and its blocking decisions resolved first. + +## Dependency graph + +```text +approve spec + resolve platform versions/host/runner/backup decisions + | + +--> release manifest + configuration/secret schemas + | | + | +--> Compose networks/data services + | | | + | | +--> migration/import/admin lifecycle + | | +--> API/worker/frontend services + | | + | +--> image/build contracts in service repositories + | | + | +--> cross-repository candidate events + | + +--> Nginx/TLS/routing contract implementation + | + +--> observability baseline + | + +--> backup --> guarded restore --> restore rehearsal + | + +--> host bootstrap validation + | + v + full release/rollback/DR acceptance +``` + +## Architecture decisions + +- Preserve ADR 0001's three-repository topology. +- Build application images once in service CI and deploy immutable digests through ADR 0002 release manifests. +- Consume the licensed font/license through the ordinary frontend context per accepted ADR 0003 and frontend commit `64f5ac2`; no deployment injection path. +- Keep PostgreSQL authoritative and backed up. Redis uses routine AOF persistence but remains excluded from disaster-recovery backup. +- Separate API, worker, migration, one-time import, and administrative commands. No recurring CSV or scheduled application worker exists. +- Use expand/contract migrations and application-image rollback; never automatic database down migrations. +- Keep Nginx as the only public entry point and monitoring/operator access private. + +## Phases and checkpoints + +### Phase 0: approval and evidence + +- Resolve host/provider/capacity, container privilege model, forge/runner/registry versions, exact image versions/digests, resource budgets, API health/routes, backup encryption/provider, certificate-renewal method, and operator ownership. +- Approve `docs/PLATFORM-OPERATIONS-SPEC.md` and ADR 0002. ADR 0003 is already accepted. +- Obtain approved machine-readable backend/frontend interface and runtime command contracts. + +Checkpoint: no implementation file exists before the approval record and decisions are complete. + +### Phase 1: release and configuration foundation + +- Define and validate the release-manifest schema, active/previous release records, exact three-repository/image provenance, and compatibility fields. +- Define tracked safe configuration examples plus the external host secret-file contract. +- Add policy tests that reject floating images, unintended ports, secret-like values, missing health checks, or font injection paths. + +Checkpoint: a synthetic manifest/config validates offline and contains no secret values or mutable references. + +### Phase 2: runtime topology and edge + +- Implement isolated Compose networks, named volumes, PostgreSQL/Redis health/resources, private application services, and monitoring profile. +- Implement one-off migration, import, and administrative profiles without coupling them to normal startup. +- Implement canonical-domain Nginx/TLS, SPA/API separation, exact SSE route behavior, security headers, limits, and health routing. +- Add focused topology/routing tests before deployment automation. + +Checkpoint: local/integration inspection proves only Nginx publishes 80/443, startup gates are correct, and SSE is not buffered. + +### Phase 3: service artifact and lifecycle coordination + +- Coordinate backend image commands/health/labels and frontend static/font health with the owning repositories through approved changes there; do not duplicate their source here. +- Implement deployment locks, preflight, migration, service replacement, postflight, observation, and application rollback using release manifests. +- Implement Redis-loss reconciliation acceptance with backend/worker owners, preserving Asynq `v0.26.0` as a pinned pre-1.0 dependency. + +Checkpoint: an induced migration/readiness/font/worker failure preserves or restores the prior release. + +### Phase 4: CI/CD and provenance + +- Add deployment-repository validation and production-release workflows. +- Coordinate service-repository workflows that publish immutable images and send candidate events only after required checks. +- Enforce protected environment approval, serialized release, registry digest retention, SBOM/provenance/vulnerability gates, and secret isolation. + +Checkpoint: PRs never deploy; backend/frontend candidate events cannot directly mutate production; one manifest controls a tested release and rollback. + +### Phase 5: observability and recovery + +- Implement bounded structured container logging, Prometheus/exporters/Grafana, dashboards, and symptom alerts with runbook links. +- Implement encrypted logical PostgreSQL backup, remote verification, three-local/seven-remote retention, and telemetry. +- Implement exact-destination guarded restore into an empty destination and application validation. +- Rehearse restore and record achieved RPO/RTO. + +Checkpoint: induced failures are diagnosable from telemetry, backup age alert fires, and clean restore succeeds within two hours. + +### Phase 6: bootstrap and full acceptance + +- Convert the reviewed first-server checklist into only the minimum approved automation, retaining manual safety/identity gates. +- Bootstrap the selected host, stage DNS/TLS, deploy a release, run one-time import/credentials, and complete alert/backup/restore handoff. +- Execute every measurable criterion from the Platform Operations spec and retain non-sensitive evidence. + +Checkpoint: stakeholder review confirms release, rollback, Redis-loss, backup/restore, exposure, SLO, and operator-boundary evidence. + +## Risks and mitigations + +| Risk | Impact | Mitigation | +|---|---|---| +| Single-host failure | Full outage | Immutable rebuild path, encrypted off-host PostgreSQL backup, two-hour restore rehearsal, documented DNS/host replacement | +| Cross-repository race | Incompatible release | Candidate events only, serialized deployment lock, exact manifest, compatibility metadata, one coordinator | +| Migration blocks rollback | Extended outage/data risk | Expand/contract, measured locks, fresh verified backup, no automatic down migration | +| Redis loss drops in-flight queue/cache | Partial analytics interruption | AOF for routine restart, PostgreSQL authority, idempotent stale-run reconciliation, cold-cache tests | +| SSE proxy buffering/timeouts | Dashboard appears stuck | Exact route, buffering/compression/cache off, heartbeat-aware timeout, integration test | +| Docker privilege compromise | Host root compromise | Approve rootless/dedicated model, restricted machine identity/SSH, no daemon TCP exposure, minimal host users | +| Secrets leak through Compose/log/backup | Credential/data compromise | File-mounted least privilege, redaction tests, no env in backup, rotate-first incident rule | +| Backup exists but is unusable | Data loss beyond RTO | Archive/checksum/remote verification plus monthly clean restore | +| Font missing/divergent | Broken approved typography/license trail | Frontend-owned tracked inputs at `64f5ac2+`, build/static smoke, digest provenance, no second path | +| Monitoring overload/noise | Lost diagnostics or ignored alerts | Bounded labels/retention, symptom alerts, test-fire, only page/ticket severities | + +## Parallelization after contracts are fixed + +- Safe in parallel: Nginx focused tests, backup/restore tooling, observability dashboards, and CI validation, provided the shared manifest/config schemas are merged first. +- Sequential: schema migrations, production deploy/rollback, database restore, DNS/TLS cutover, and one-time import. +- Coordinated across repositories: backend commands/health/migration metadata, frontend image/font health, and candidate-event payload. Providers define contracts before consumers implement them. + +## Completion gate + +- Every task in `tasks/todo.md` meets acceptance and verification evidence. +- All spec acceptance criteria pass on a production-shaped host. +- No unresolved decision remains hidden in a default or script. +- Active/previous release manifests, SLO dashboards, alerts, backup, restore rehearsal, and operator handoff are complete. +- Backend/frontend repositories contain only their approved coordinated changes; the deployment repository contains no application source, raw data, secrets, credentials, TLS keys, backups, database volumes, or duplicate font/license files. diff --git a/tasks/todo.md b/tasks/todo.md new file mode 100644 index 0000000..0479283 --- /dev/null +++ b/tasks/todo.md @@ -0,0 +1,243 @@ +# Platform Operations implementation tasks + +- Status: Not authorized — wait for explicit approval of the draft spec +- Canonical plan: `tasks/plan.md` + +## Task 0: Approve specification and resolve blocking decisions + +**Description:** Review the platform spec/ADRs and record every host, runner, version, route, capacity, backup, TLS, and ownership decision before implementation. + +**Acceptance criteria:** +- [ ] `PLATFORM-OPERATIONS-SPEC.md` status is explicitly approved by the stakeholder. +- [ ] ADR 0002 is accepted or replaced; ADR 0003 remains consistent with frontend commit `64f5ac2`. +- [ ] All ten remaining decisions have named answers/evidence. + +**Verification:** Manual stakeholder approval and clean documentation diff. + +**Dependencies:** None + +**Files likely touched:** `docs/PLATFORM-OPERATIONS-SPEC.md`, `docs/decisions/0002-immutable-cross-repository-releases.md` + +**Estimated scope:** Small (1–2 files) + +## Task 1: Define release manifest and safe configuration schemas + +**Description:** Create machine-validated schemas/examples for exact release provenance, compatibility metadata, active/previous pointers, safe public configuration, and external secret files. + +**Acceptance criteria:** +- [ ] Synthetic valid/invalid manifests prove immutable three-repository/image constraints. +- [ ] Examples contain no secret-like values and name every required configuration input. +- [ ] Font provenance is only frontend SHA/image digest; no injection field/path exists. + +**Verification:** Schema tests plus secret scan over staged files. + +**Dependencies:** Task 0 + +**Files likely touched:** release/config schemas, safe examples, focused tests (maximum 5 files) + +**Estimated scope:** Medium (3–5 files) + +## Task 2: Implement data-service Compose foundation + +**Description:** Add PostgreSQL/Redis pinned services, isolated data network, volumes, authentication, health checks, AOF policy, resources, and no host port publishing. + +**Acceptance criteria:** +- [ ] Compose renders with pinned digests and healthy PostgreSQL/Redis. +- [ ] Redis loss is explicitly non-authoritative and its volume is excluded from backup. +- [ ] External/local port inspection finds no PostgreSQL/Redis host listener. + +**Verification:** Compose validation, health inspection, network/port policy tests. + +**Dependencies:** Task 1 + +**Files likely touched:** Compose files and focused topology tests (maximum 5 files) + +**Estimated scope:** Medium (3–5 files) + +## Task 3: Implement API, worker, and one-off lifecycle boundaries + +**Description:** Wire immutable backend API/worker plus migration/import/admin one-offs with health gates, profiles, locks, graceful stop, and validated configuration. + +**Acceptance criteria:** +- [ ] Normal startup waits for migration but never runs import/admin automatically. +- [ ] Worker is private/event-driven and retains pinned pre-1.0 Asynq `v0.26.0` contract. +- [ ] Failed migration blocks candidate API/worker; import path is exact/read-only/idempotent. + +**Verification:** Lifecycle integration tests including failed migration and repeated import fixture. + +**Dependencies:** Tasks 1–2 and approved backend command/health contract + +**Files likely touched:** Compose lifecycle definitions, wrapper, focused tests (maximum 5 files) + +**Estimated scope:** Medium (3–5 files) + +## Task 4: Implement frontend and font artifact health + +**Description:** Consume the ordinary frontend image/build context and verify static SPA plus tracked font output without copying/injecting assets in deployment. + +**Acceptance criteria:** +- [ ] Candidate maps to frontend SHA/digest and serves expected font URL/MIME. +- [ ] Missing font/license input fails frontend CI or candidate smoke. +- [ ] Deployment files contain no font binary, license copy, private fetch, or font secret. + +**Verification:** Image/static health test and repository policy scan. + +**Dependencies:** Task 1 and approved frontend image contract at/after `64f5ac2` + +**Files likely touched:** frontend service definition and focused smoke/policy tests (maximum 5 files) + +**Estimated scope:** Small/Medium (2–4 files) + +## Task 5: Implement Nginx canonical TLS and routing + +**Description:** Add canonical host/default rejection, ACME/TLS, API/frontend separation, exact SSE proxy behavior, security headers, limits, and generic health routes. + +**Acceptance criteria:** +- [ ] Only 80/443 are public; redirects/default-host behavior is exact. +- [ ] SSE buffering/cache/compression are off and heartbeat/replay survives reconnect. +- [ ] API never falls through to SPA; security/TLS/static-cache tests pass. + +**Verification:** `nginx -t`, TLS/header/routing tests, streamed timing/reconnect test. + +**Dependencies:** Tasks 3–4 and approved API/SSE paths/CSP + +**Files likely touched:** Nginx template/entrypoint and focused tests (maximum 5 files) + +**Estimated scope:** Medium (3–5 files) + +## Checkpoint A: Runtime foundation + +- [ ] Tasks 1–5 pass together from a clean Compose project. +- [ ] Only Nginx is public and all health/startup boundaries behave as specified. +- [ ] No secret, raw data, backup, TLS key, or duplicate font/license asset is tracked. + +## Task 6: Implement release preflight, deploy, and rollback + +**Description:** Build the serialized manifest-driven operator workflow for preflight, migration, service replacement, postflight, observation, active/previous switch, and rollback. + +**Acceptance criteria:** +- [ ] Deploy uses image digests only and records evidence without secrets. +- [ ] Candidate health/migration/font failure preserves/restores prior release. +- [ ] Application-only rollback completes within 15 minutes and never auto-runs down migration. + +**Verification:** Shell/unit tests plus induced staging failures and timed rollback drill. + +**Dependencies:** Checkpoint A + +**Files likely touched:** deploy/release scripts and focused tests (maximum 5 files per incremental slice) + +**Estimated scope:** Split into multiple Medium tasks during implementation + +## Task 7: Implement deployment-repository CI/CD + +**Description:** Validate docs/config/Compose/Nginx/scripts/policies, accept authenticated candidates, enforce protected approval/locking, and invoke the manifest-driven release. + +**Acceptance criteria:** +- [ ] PR workflows validate only and cannot access production secrets/deploy. +- [ ] Production release requires gates plus human approval and is serialized. +- [ ] Provenance/SBOM/scan/secret results attach to the release record. + +**Verification:** Forge event tests with denied PR deploy, concurrent candidate events, failed gate, and approved release. + +**Dependencies:** Task 6 and confirmed forge/runner/registry versions + +**Files likely touched:** workflow plus CI-focused helpers/tests (maximum 5 files) + +**Estimated scope:** Medium (3–5 files) + +## Task 8: Coordinate backend/frontend candidate workflows + +**Description:** In separately approved service-repository changes, publish one immutable tested image and send candidate metadata without direct production SSH. + +**Acceptance criteria:** +- [ ] Backend gates include Go/migration/import/Asynq queue/recovery verification. +- [ ] Frontend gates include Svelte/accessibility/RTL/build/font verification. +- [ ] Service events cannot mutate production and use least-privilege credentials. + +**Verification:** Each repository's CI on PR/main plus deployment candidate receipt test. + +**Dependencies:** Task 7 and separate authorization in each service repository + +**Files likely touched:** owned service workflow/tests only; not this deployment task + +**Estimated scope:** Separate cross-repository tasks + +## Task 9: Implement observability baseline + +**Description:** Add bounded structured logging, private metrics/exporters/dashboards, release correlation, and symptom alerts/runbook links. + +**Acceptance criteria:** +- [ ] All required RED/USE/worker/SSE/backup/TLS signals exist with bounded labels. +- [ ] Logs rotate and contain request/release correlation but no forbidden fields. +- [ ] Page/ticket alerts are test-fired and monitoring is not public. + +**Verification:** Induced staging failure located from telemetry, cardinality/redaction checks, alert delivery test. + +**Dependencies:** Checkpoint A and approved monitoring versions/budgets + +**Files likely touched:** Compose monitoring slice/config/dashboards/tests, split to <=5 files per commit + +**Estimated scope:** Split into multiple Medium tasks + +## Task 10: Implement encrypted backup and retention + +**Description:** Produce verified logical PostgreSQL backups, encrypt before durable storage, upload/verify remotely, retain three local/seven remote, and emit metrics. + +**Acceptance criteria:** +- [ ] Archive/listing/checksums/manifest/remote verification gate success. +- [ ] Backup contains none of the explicit exclusions. +- [ ] Cleanup never removes the last known-good copy and uses exact validated paths. + +**Verification:** Backup tests against fixture DB/object store, archive exclusion inspection, failed-upload/cleanup tests. + +**Dependencies:** Tasks 1–3 and approved encryption/provider decisions + +**Files likely touched:** backup script/config/tests/docs (maximum 5 files per slice) + +**Estimated scope:** Medium (3–5 files) + +## Task 11: Implement guarded restore and rehearsal + +**Description:** Validate/decrypt/trust-check an explicit backup, require all production guards, restore into an empty destination, and run application validation. + +**Acceptance criteria:** +- [ ] Each absent/mismatched production factor aborts before mutation. +- [ ] Partial/failed restore is never treated as usable or blindly retried. +- [ ] Clean rehearsal passes tenant/session/cold-cache/worker/SSE checks within two hours. + +**Verification:** Destructive tests only against exact disposable fixtures, production-guard negative matrix, timed rehearsal. + +**Dependencies:** Task 10 and approved switch strategy + +**Files likely touched:** restore script/tests/docs (maximum 5 files per slice) + +**Estimated scope:** Medium (3–5 files) + +## Checkpoint B: Release and recovery + +- [ ] Tasks 6–11 pass in staging with release/rollback/backup/restore evidence. +- [ ] Registry, runner, monitoring, object storage, and secrets remain least privilege/private. + +## Task 12: Bootstrap and accept first production-shaped server + +**Description:** Follow the reviewed bootstrap runbook, deploy an approved manifest, run the one-time data/demo setup, and capture every acceptance criterion. + +**Acceptance criteria:** +- [ ] Access/firewall/Docker/DNS/TLS/deploy/monitoring boundaries pass. +- [ ] Migration/import/demo accounts/tenant isolation/analytics SSE pass. +- [ ] Encrypted remote backup and clean restore rehearsal meet RPO/RTO. + +**Verification:** Completed runbook evidence and stakeholder acceptance review. + +**Dependencies:** Checkpoint B and approved host/change window + +**Files likely touched:** evidence/release records only; no secret/raw runtime artifacts + +**Estimated scope:** Operational session + +## Final checkpoint + +- [ ] Every measurable acceptance criterion in the approved spec passes. +- [ ] Active and previous releases are exact and rollback-ready. +- [ ] Remaining decisions are resolved or explicitly defer launch. +- [ ] Documentation reflects implemented reality and no implementation gate was bypassed.