# Implementation Plan: Platform Operations - Status: Planned from draft revision; implementation blocked pending explicit spec approval - Capability: Platform operations - Task list target: `tasks/todo.md` - Last updated: 2026-08-20 ## Confirmed implementation direction Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds. There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work. No task below is authorized merely because it appears here. ## Dependency graph ```text approve revised spec + resolve server/runtime contracts | +--> safe config + manual release-record contract | | | +--> Compose data/app services | | | | | +--> migrate/import/admin separation | | +--> manual deploy/rollback | | | +--> Nginx/Certbot routing and TLS | +--> Qlockify backup adaptation --> guarded restore --> rehearsal | +--> first-server bootstrap --> MVP acceptance | v Prometheus/Grafana phase | v AI may begin ``` ## Architecture decisions - Preserve ADR 0001's three independent repositories and ignored nested service checkouts. - Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records. - Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license. - Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags. - Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup. - Separate API, worker, migration, one-time CSV import, and administrative processes. - Keep Asynq pinned to pre-1.0 `v0.26.0`. - Use automatic Let's Encrypt/Certbot TLS. - Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI. ## Phase 0: approve exact contracts - Approve the revised Platform Operations spec. - Resolve server/provider/capacity/user/path, image tags, resource limits, repository URLs/commands/health routes, backup endpoint/credentials/schedule, Redis AOF choice, release-record format, operator/secret escrow, and MVP-ready/observability gates. - Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery. - Obtain the frontend build/static health contract. Checkpoint: no implementation file exists before explicit approval and resolved inputs. ## Phase 1: configuration and Compose foundation - Add safe tracked configuration examples and validate protected external secret paths. - Define non-secret active/previous release records containing the three SHAs and health/migration evidence. - Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles. - Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts. Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin. ## Phase 2: Nginx, TLS, and lifecycle - Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`. - Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior. - Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset. - Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release. Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes. ## Phase 3: backup, restore, and first-server acceptance - Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials. - Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only. - Add exact destination and production confirmation guards. - Bootstrap the server, deploy the MVP, perform one-time import/credential setup, create an encrypted remote backup, and rehearse clean restore within two hours. Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks. ## Phase 4: required observability before AI - Add Prometheus, Grafana, and minimum private exporters/instrumentation. - Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards. - Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery. - Validate the observability acceptance gate before any AI implementation starts. Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator. ## Risks and mitigations | Risk | Impact | Mitigation | |---|---|---| | Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record | | Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` | | Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time | | Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration | | Redis loss drops queue/cache | Partial analytics interruption | Optional AOF, PostgreSQL authority, idempotent reconciliation/cold-cache tests | | SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test | | Docker-group compromise | Host-root compromise | Dedicated key-only trusted account, restricted SSH, no daemon TCP exposure | | Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests | | Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal | | Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI | ## Parallelization after approval - Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed. - Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate. - Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced. ## Completion - Every authorized task in `tasks/todo.md` passes its acceptance and verification. - The active stack maps to exact deployment/backend/frontend commits. - Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass. - Prometheus/Grafana acceptance passes before AI begins. - Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets.