126 lines
8.0 KiB
Markdown
126 lines
8.0 KiB
Markdown
# Implementation Plan: Platform Operations
|
|
|
|
- Status: Planned from draft revision; implementation blocked pending explicit spec approval
|
|
- Capability: Platform operations
|
|
- Task list target: `tasks/todo.md`
|
|
- Last updated: 2026-08-20
|
|
|
|
## Confirmed implementation direction
|
|
|
|
Build a Qlockify-style, single-Ubuntu-server platform for `zarinpal.amiirkhl.ir`. A trusted Docker-group deployment user manually pulls the latest `main` in the deployment, backend, and frontend repositories; records all three commit hashes; builds locally with Docker Compose; migrates; verifies health; and records the release. Rollback restores the previous hashes and rebuilds.
|
|
|
|
There is no CI/CD pipeline, runner, private image registry, or automated cross-repository trigger. PostgreSQL backup/restore adapts Qlockify's existing scripts with minimal project changes and excludes environment/secrets. Prometheus/Grafana are a required second phase after MVP readiness and before AI work.
|
|
|
|
No task below is authorized merely because it appears here.
|
|
|
|
## Dependency graph
|
|
|
|
```text
|
|
approve revised spec + resolve server/runtime contracts
|
|
|
|
|
+--> safe config + manual release-record contract
|
|
| |
|
|
| +--> Compose data/app services
|
|
| | |
|
|
| | +--> migrate/import/admin separation
|
|
| | +--> manual deploy/rollback
|
|
| |
|
|
| +--> Nginx/Certbot routing and TLS
|
|
|
|
|
+--> Qlockify backup adaptation --> guarded restore --> rehearsal
|
|
|
|
|
+--> first-server bootstrap --> MVP acceptance
|
|
|
|
|
v
|
|
Prometheus/Grafana phase
|
|
|
|
|
v
|
|
AI may begin
|
|
```
|
|
|
|
## Architecture decisions
|
|
|
|
- Preserve ADR 0001's three independent repositories and ignored nested service checkouts.
|
|
- Follow accepted ADR 0002: latest `main`, record three hashes, build manually on the server, store active/previous release records.
|
|
- Follow accepted ADR 0003 and frontend commit `64f5ac2`: ordinary frontend context only for font/license.
|
|
- Use PostgreSQL 18 and Redis 7 Alpine with explicit supported tags.
|
|
- Keep PostgreSQL authoritative and backed up. Redis remains excluded from disaster-recovery backup.
|
|
- Separate API, worker, migration, one-time CSV import, and administrative processes.
|
|
- Keep Asynq pinned to pre-1.0 `v0.26.0`.
|
|
- Use automatic Let's Encrypt/Certbot TLS.
|
|
- Treat structured logs/health/manual checks as MVP; add Prometheus/Grafana after MVP and before AI.
|
|
|
|
## Phase 0: approve exact contracts
|
|
|
|
- Approve the revised Platform Operations spec.
|
|
- Resolve server/provider/capacity/user/path, image tags, resource limits, repository URLs/commands/health routes, backup endpoint/credentials/schedule, Redis AOF choice, release-record format, operator/secret escrow, and MVP-ready/observability gates.
|
|
- Obtain the backend command contract for API, worker, migration, import, admin, liveness, readiness, and Redis-loss recovery.
|
|
- Obtain the frontend build/static health contract.
|
|
|
|
Checkpoint: no implementation file exists before explicit approval and resolved inputs.
|
|
|
|
## Phase 1: configuration and Compose foundation
|
|
|
|
- Add safe tracked configuration examples and validate protected external secret paths.
|
|
- Define non-secret active/previous release records containing the three SHAs and health/migration evidence.
|
|
- Implement isolated Compose networks, PostgreSQL/Redis volumes/health/resources, backend/worker/frontend services, and one-off migration/import/admin profiles.
|
|
- Add focused validation for ports, startup ordering, font build presence, and forbidden tracked/runtime artifacts.
|
|
|
|
Checkpoint: clean local/server validation proves only Nginx will publish ports and normal startup cannot trigger import/admin.
|
|
|
|
## Phase 2: Nginx, TLS, and lifecycle
|
|
|
|
- Adapt Qlockify Nginx/Certbot structure for `zarinpal.amiirkhl.ir`.
|
|
- Implement canonical redirect/default-host rejection, API/frontend separation, exact SSE route, security headers, caching, and health behavior.
|
|
- Implement manual deploy/release/rollback from Qlockify's script structure without unconditional destructive Git reset.
|
|
- Exercise failed pull/build/migration/readiness/font/worker conditions while preserving the active release.
|
|
|
|
Checkpoint: manual deployment records the three SHAs, passes TLS/health/SSE checks, and a prior-hash rebuild rollback completes within 15 minutes.
|
|
|
|
## Phase 3: backup, restore, and first-server acceptance
|
|
|
|
- Minimally adapt Qlockify backup/upload/restore/S3-restore scripts for ZarinPal paths/services/credentials.
|
|
- Remove Qlockify environment-file backup/restore; include PostgreSQL archive plus non-secret manifest/checksums only.
|
|
- Add exact destination and production confirmation guards.
|
|
- Bootstrap the server, deploy the MVP, perform one-time import/credential setup, create an encrypted remote backup, and rehearse clean restore within two hours.
|
|
|
|
Checkpoint: every MVP acceptance criterion passes through manual health/log/disk/certificate/queue/backup checks.
|
|
|
|
## Phase 4: required observability before AI
|
|
|
|
- Add Prometheus, Grafana, and minimum private exporters/instrumentation.
|
|
- Add service/release, worker/SSE, PostgreSQL/Redis, host/container, backup/restore, and TLS dashboards.
|
|
- Add bounded labels, retention, symptom-based alerts, runbook links, and tested notification delivery.
|
|
- Validate the observability acceptance gate before any AI implementation starts.
|
|
|
|
Checkpoint: an induced failure is detected, correlated to the three-hash release, diagnosed from telemetry, and alerts reach the responsible operator.
|
|
|
|
## Risks and mitigations
|
|
|
|
| Risk | Impact | Mitigation |
|
|
|---|---|---|
|
|
| Manual deploy selects inconsistent source | Broken stack | One deployment lock, latest three `main` checkouts recorded before build, one release record |
|
|
| Dirty checkout is destroyed | Lost operator/user work | Fail on dirty/divergent state; no unconditional `git reset --hard` |
|
|
| Server-side build differs over time | Harder reproduction | Pin base/dependency inputs in service repos, record Git hashes/local image ids/build time |
|
|
| Migration blocks rollback | Extended outage | Expand/contract, measured locks, fresh backup, no automatic down migration |
|
|
| Redis loss drops queue/cache | Partial analytics interruption | Optional AOF, PostgreSQL authority, idempotent reconciliation/cold-cache tests |
|
|
| SSE buffering/timeouts | Dashboard appears stuck | Exact Nginx route, buffering/compression/cache off, heartbeat/reconnect test |
|
|
| Docker-group compromise | Host-root compromise | Dedicated key-only trusted account, restricted SSH, no daemon TCP exposure |
|
|
| Backup leaks credentials | Wider compromise | Remove `.env` backup/restore, encrypt, separate passphrase, archive exclusion tests |
|
|
| Backup is unusable | Data loss beyond RTO | Checksums/listing/remote verification plus clean restore rehearsal |
|
|
| Delayed monitoring hides degradation | Slower MVP incident diagnosis | MVP health/log/operator checklist; mandatory Prometheus/Grafana before AI |
|
|
|
|
## Parallelization after approval
|
|
|
|
- Safe in parallel: focused Nginx tests, backup adaptation, release-record validation, and later dashboards after shared paths/contracts are fixed.
|
|
- Sequential: deployment, migration, production restore, DNS/TLS cutover, one-time import, and MVP-to-observability phase gate.
|
|
- Cross-repository coordination: backend commands/health and frontend build/font behavior are defined by their owners before deployment consumes them; no automated pipeline is introduced.
|
|
|
|
## Completion
|
|
|
|
- Every authorized task in `tasks/todo.md` passes its acceptance and verification.
|
|
- The active stack maps to exact deployment/backend/frontend commits.
|
|
- Rollback, Redis-loss recovery, backup, restore, import separation, and exposure checks pass.
|
|
- Prometheus/Grafana acceptance passes before AI begins.
|
|
- Deployment Git contains no application source, secrets, raw data, credentials, certificates, backups, volumes, or duplicate font/license assets.
|