# Implementation Plan: Platform Operations - Status: Planned from draft specification; implementation blocked pending explicit approval - Capability: Platform operations - Task list target: `tasks/todo.md` - Last updated: 2026-08-20 ## Overview Implement the approved Platform Operations contract as a single-server, production-shaped Docker Compose platform for `zarinpal.amiirkhl.ir`. The deployment repository remains the only operational coordinator; backend and frontend source stay in their own repositories and publish immutable images. Work proceeds from decisions and validation contracts through topology, edge, lifecycle, release automation, recovery, and end-to-end proof. No task below is authorized merely because it appears in this plan. The Platform Operations spec must be approved and its blocking decisions resolved first. ## Dependency graph ```text approve spec + resolve platform versions/host/runner/backup decisions | +--> release manifest + configuration/secret schemas | | | +--> Compose networks/data services | | | | | +--> migration/import/admin lifecycle | | +--> API/worker/frontend services | | | +--> image/build contracts in service repositories | | | +--> cross-repository candidate events | +--> Nginx/TLS/routing contract implementation | +--> observability baseline | +--> backup --> guarded restore --> restore rehearsal | +--> host bootstrap validation | v full release/rollback/DR acceptance ``` ## Architecture decisions - Preserve ADR 0001's three-repository topology. - Build application images once in service CI and deploy immutable digests through ADR 0002 release manifests. - Consume the licensed font/license through the ordinary frontend context per accepted ADR 0003 and frontend commit `64f5ac2`; no deployment injection path. - Keep PostgreSQL authoritative and backed up. Redis uses routine AOF persistence but remains excluded from disaster-recovery backup. - Separate API, worker, migration, one-time import, and administrative commands. No recurring CSV or scheduled application worker exists. - Use expand/contract migrations and application-image rollback; never automatic database down migrations. - Keep Nginx as the only public entry point and monitoring/operator access private. ## Phases and checkpoints ### Phase 0: approval and evidence - Resolve host/provider/capacity, container privilege model, forge/runner/registry versions, exact image versions/digests, resource budgets, API health/routes, backup encryption/provider, certificate-renewal method, and operator ownership. - Approve `docs/PLATFORM-OPERATIONS-SPEC.md` and ADR 0002. ADR 0003 is already accepted. - Obtain approved machine-readable backend/frontend interface and runtime command contracts. Checkpoint: no implementation file exists before the approval record and decisions are complete. ### Phase 1: release and configuration foundation - Define and validate the release-manifest schema, active/previous release records, exact three-repository/image provenance, and compatibility fields. - Define tracked safe configuration examples plus the external host secret-file contract. - Add policy tests that reject floating images, unintended ports, secret-like values, missing health checks, or font injection paths. Checkpoint: a synthetic manifest/config validates offline and contains no secret values or mutable references. ### Phase 2: runtime topology and edge - Implement isolated Compose networks, named volumes, PostgreSQL/Redis health/resources, private application services, and monitoring profile. - Implement one-off migration, import, and administrative profiles without coupling them to normal startup. - Implement canonical-domain Nginx/TLS, SPA/API separation, exact SSE route behavior, security headers, limits, and health routing. - Add focused topology/routing tests before deployment automation. Checkpoint: local/integration inspection proves only Nginx publishes 80/443, startup gates are correct, and SSE is not buffered. ### Phase 3: service artifact and lifecycle coordination - Coordinate backend image commands/health/labels and frontend static/font health with the owning repositories through approved changes there; do not duplicate their source here. - Implement deployment locks, preflight, migration, service replacement, postflight, observation, and application rollback using release manifests. - Implement Redis-loss reconciliation acceptance with backend/worker owners, preserving Asynq `v0.26.0` as a pinned pre-1.0 dependency. Checkpoint: an induced migration/readiness/font/worker failure preserves or restores the prior release. ### Phase 4: CI/CD and provenance - Add deployment-repository validation and production-release workflows. - Coordinate service-repository workflows that publish immutable images and send candidate events only after required checks. - Enforce protected environment approval, serialized release, registry digest retention, SBOM/provenance/vulnerability gates, and secret isolation. Checkpoint: PRs never deploy; backend/frontend candidate events cannot directly mutate production; one manifest controls a tested release and rollback. ### Phase 5: observability and recovery - Implement bounded structured container logging, Prometheus/exporters/Grafana, dashboards, and symptom alerts with runbook links. - Implement encrypted logical PostgreSQL backup, remote verification, three-local/seven-remote retention, and telemetry. - Implement exact-destination guarded restore into an empty destination and application validation. - Rehearse restore and record achieved RPO/RTO. Checkpoint: induced failures are diagnosable from telemetry, backup age alert fires, and clean restore succeeds within two hours. ### Phase 6: bootstrap and full acceptance - Convert the reviewed first-server checklist into only the minimum approved automation, retaining manual safety/identity gates. - Bootstrap the selected host, stage DNS/TLS, deploy a release, run one-time import/credentials, and complete alert/backup/restore handoff. - Execute every measurable criterion from the Platform Operations spec and retain non-sensitive evidence. Checkpoint: stakeholder review confirms release, rollback, Redis-loss, backup/restore, exposure, SLO, and operator-boundary evidence. ## Risks and mitigations | Risk | Impact | Mitigation | |---|---|---| | Single-host failure | Full outage | Immutable rebuild path, encrypted off-host PostgreSQL backup, two-hour restore rehearsal, documented DNS/host replacement | | Cross-repository race | Incompatible release | Candidate events only, serialized deployment lock, exact manifest, compatibility metadata, one coordinator | | Migration blocks rollback | Extended outage/data risk | Expand/contract, measured locks, fresh verified backup, no automatic down migration | | Redis loss drops in-flight queue/cache | Partial analytics interruption | AOF for routine restart, PostgreSQL authority, idempotent stale-run reconciliation, cold-cache tests | | SSE proxy buffering/timeouts | Dashboard appears stuck | Exact route, buffering/compression/cache off, heartbeat-aware timeout, integration test | | Docker privilege compromise | Host root compromise | Approve rootless/dedicated model, restricted machine identity/SSH, no daemon TCP exposure, minimal host users | | Secrets leak through Compose/log/backup | Credential/data compromise | File-mounted least privilege, redaction tests, no env in backup, rotate-first incident rule | | Backup exists but is unusable | Data loss beyond RTO | Archive/checksum/remote verification plus monthly clean restore | | Font missing/divergent | Broken approved typography/license trail | Frontend-owned tracked inputs at `64f5ac2+`, build/static smoke, digest provenance, no second path | | Monitoring overload/noise | Lost diagnostics or ignored alerts | Bounded labels/retention, symptom alerts, test-fire, only page/ticket severities | ## Parallelization after contracts are fixed - Safe in parallel: Nginx focused tests, backup/restore tooling, observability dashboards, and CI validation, provided the shared manifest/config schemas are merged first. - Sequential: schema migrations, production deploy/rollback, database restore, DNS/TLS cutover, and one-time import. - Coordinated across repositories: backend commands/health/migration metadata, frontend image/font health, and candidate-event payload. Providers define contracts before consumers implement them. ## Completion gate - Every task in `tasks/todo.md` meets acceptance and verification evidence. - All spec acceptance criteria pass on a production-shaped host. - No unresolved decision remains hidden in a default or script. - Active/previous release manifests, SLO dashboards, alerts, backup, restore rehearsal, and operator handoff are complete. - Backend/frontend repositories contain only their approved coordinated changes; the deployment repository contains no application source, raw data, secrets, credentials, TLS keys, backups, database volumes, or duplicate font/license files.