127 lines
8.2 KiB
Markdown
127 lines
8.2 KiB
Markdown
# Implementation Plan: Platform Operations
|
||
|
||
- Status: Implementation plan ready for stakeholder review — implementation not authorized
|
||
- Capability: Platform operations
|
||
- Task list target: `tasks/todo.md`
|
||
- Last updated: 2026-08-20
|
||
|
||
## Confirmed implementation direction
|
||
|
||
Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds.
|
||
|
||
There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work.
|
||
|
||
No task below is authorized merely because it appears here.
|
||
|
||
## Dependency graph
|
||
|
||
```text
|
||
resolve concrete server/runtime inputs + authorize build
|
||
|
|
||
+--> safe config + manual release-record contract
|
||
| |
|
||
| +--> Compose data/app services
|
||
| | |
|
||
| | +--> migrate/import/admin separation
|
||
| | +--> manual deploy/rollback
|
||
| |
|
||
| +--> Nginx/Certbot routing and TLS
|
||
|
|
||
+--> Qlockify backup adaptation --> guarded restore --> rehearsal
|
||
|
|
||
+--> first-server bootstrap --> MVP acceptance
|
||
|
|
||
v
|
||
Prometheus/Grafana phase
|
||
|
|
||
v
|
||
AI may begin
|
||
```
|
||
|
||
## Architecture decisions
|
||
|
||
- Preserve ADR 0001's three independent repositories and ignored nested service checkouts.
|
||
- Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records.
|
||
- Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license.
|
||
- Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags.
|
||
- Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup.
|
||
- Separate API, worker, migration, one-time CSV import, and administrative processes.
|
||
- Keep Asynq pinned to pre-1.0 `v0.26.0`.
|
||
- Use automatic Let's Encrypt/Certbot TLS.
|
||
- Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI.
|
||
|
||
## Phase 0: resolve implementation inputs and authorize the build
|
||
|
||
- Treat the approved Platform Operations spec and ADRs 0001–0003 as the implementation baseline.
|
||
- Obtain the production IP/DNS state, repository URLs, actual protected environment values, ParsPack endpoint/bucket/prefix/credentials, and named operator/secret-escrow contacts.
|
||
- Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery.
|
||
- Obtain the frontend build/static health contract.
|
||
- Obtain explicit authorization before creating deployment implementation artifacts.
|
||
|
||
Checkpoint: no implementation file exists before explicit build authorization and the inputs needed by its slice are resolved.
|
||
|
||
## Phase 1: configuration and Compose foundation
|
||
|
||
- Add safe tracked configuration examples and validate protected external secret paths.
|
||
- Define non-secret active/previous release records containing the three SHAs and health/migration evidence.
|
||
- Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles.
|
||
- Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts.
|
||
|
||
Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin.
|
||
|
||
## Phase 2: Nginx, TLS, and lifecycle
|
||
|
||
- Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`.
|
||
- Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior.
|
||
- Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset.
|
||
- Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release.
|
||
|
||
Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes.
|
||
|
||
## Phase 3: backup, restore, and first-server acceptance
|
||
|
||
- Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials.
|
||
- Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only.
|
||
- Add exact destination and production confirmation guards.
|
||
- Bootstrap the server with `first-start.sh`: migrate, import/reconcile the pinned archive, then activate public services; perform separate credential setup, encrypted remote backup, and a clean restore rehearsal within two hours.
|
||
|
||
Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks.
|
||
|
||
## Phase 4: required observability before AI
|
||
|
||
- Add Prometheus, Grafana, and minimum private exporters/instrumentation.
|
||
- Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards.
|
||
- Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery.
|
||
- Validate the observability acceptance gate before any AI implementation starts.
|
||
|
||
Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator.
|
||
|
||
## Risks and mitigations
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|---|---|---|
|
||
| Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record |
|
||
| Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` |
|
||
| Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time |
|
||
| Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration |
|
||
| Redis loss drops queue/cache | Partial analytics interruption | AOF `everysec`, PostgreSQL authority, idempotent reconciliation/cold-cache tests |
|
||
| SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test |
|
||
| Docker-group compromise | Host-root compromise | Restricted key-only access to trusted `ubuntu` account, no daemon TCP exposure |
|
||
| Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests |
|
||
| Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal |
|
||
| Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI |
|
||
|
||
## Parallelization after approval
|
||
|
||
- Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed.
|
||
- Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate.
|
||
- Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced.
|
||
|
||
## Completion
|
||
|
||
- Every authorized task in `tasks/todo.md` passes its acceptance and verification.
|
||
- The active stack maps to exact deployment/backend/frontend commits.
|
||
- Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass.
|
||
- Prometheus/Grafana acceptance passes before AI begins.
|
||
- Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets.
|