140 lines
9.1 KiB
Markdown
140 lines
9.1 KiB
Markdown
# Implementation Plan: Platform Operations
|
|
|
|
- Status: Planned from draft specification; implementation blocked pending explicit approval
|
|
- Capability: Platform operations
|
|
- Task list target: `tasks/todo.md`
|
|
- Last updated: 2026-08-20
|
|
|
|
## Overview
|
|
|
|
Implement the approved Platform Operations contract as a single-server, production-shaped Docker Compose platform for `zarinpal.amiirkhl.ir`. The deployment repository remains the only operational coordinator; backend and frontend source stay in their own repositories and publish immutable images. Work proceeds from decisions and validation contracts through topology, edge, lifecycle, release automation, recovery, and end-to-end proof.
|
|
|
|
No task below is authorized merely because it appears in this plan. The Platform Operations spec must be approved and its blocking decisions resolved first.
|
|
|
|
## Dependency graph
|
|
|
|
```text
|
|
approve spec + resolve platform versions/host/runner/backup decisions
|
|
|
|
|
+--> release manifest + configuration/secret schemas
|
|
| |
|
|
| +--> Compose networks/data services
|
|
| | |
|
|
| | +--> migration/import/admin lifecycle
|
|
| | +--> API/worker/frontend services
|
|
| |
|
|
| +--> image/build contracts in service repositories
|
|
| |
|
|
| +--> cross-repository candidate events
|
|
|
|
|
+--> Nginx/TLS/routing contract implementation
|
|
|
|
|
+--> observability baseline
|
|
|
|
|
+--> backup --> guarded restore --> restore rehearsal
|
|
|
|
|
+--> host bootstrap validation
|
|
|
|
|
v
|
|
full release/rollback/DR acceptance
|
|
```
|
|
|
|
## Architecture decisions
|
|
|
|
- Preserve ADR 0001's three-repository topology.
|
|
- Build application images once in service CI and deploy immutable digests through ADR 0002 release manifests.
|
|
- Consume the licensed font/license through the ordinary frontend context per accepted ADR 0003 and frontend commit `64f5ac2`; no deployment injection path.
|
|
- Keep PostgreSQL authoritative and backed up. Redis uses routine AOF persistence but remains excluded from disaster-recovery backup.
|
|
- Separate API, worker, migration, one-time import, and administrative commands. No recurring CSV or scheduled application worker exists.
|
|
- Use expand/contract migrations and application-image rollback; never automatic database down migrations.
|
|
- Keep Nginx as the only public entry point and monitoring/operator access private.
|
|
|
|
## Phases and checkpoints
|
|
|
|
### Phase 0: approval and evidence
|
|
|
|
- Resolve host/provider/capacity, container privilege model, forge/runner/registry versions, exact image versions/digests, resource budgets, API health/routes, backup encryption/provider, certificate-renewal method, and operator ownership.
|
|
- Approve `docs/PLATFORM-OPERATIONS-SPEC.md` and ADR 0002. ADR 0003 is already accepted.
|
|
- Obtain approved machine-readable backend/frontend interface and runtime command contracts.
|
|
|
|
Checkpoint: no implementation file exists before the approval record and decisions are complete.
|
|
|
|
### Phase 1: release and configuration foundation
|
|
|
|
- Define and validate the release-manifest schema, active/previous release records, exact three-repository/image provenance, and compatibility fields.
|
|
- Define tracked safe configuration examples plus the external host secret-file contract.
|
|
- Add policy tests that reject floating images, unintended ports, secret-like values, missing health checks, or font injection paths.
|
|
|
|
Checkpoint: a synthetic manifest/config validates offline and contains no secret values or mutable references.
|
|
|
|
### Phase 2: runtime topology and edge
|
|
|
|
- Implement isolated Compose networks, named volumes, PostgreSQL/Redis health/resources, private application services, and monitoring profile.
|
|
- Implement one-off migration, import, and administrative profiles without coupling them to normal startup.
|
|
- Implement canonical-domain Nginx/TLS, SPA/API separation, exact SSE route behavior, security headers, limits, and health routing.
|
|
- Add focused topology/routing tests before deployment automation.
|
|
|
|
Checkpoint: local/integration inspection proves only Nginx publishes 80/443, startup gates are correct, and SSE is not buffered.
|
|
|
|
### Phase 3: service artifact and lifecycle coordination
|
|
|
|
- Coordinate backend image commands/health/labels and frontend static/font health with the owning repositories through approved changes there; do not duplicate their source here.
|
|
- Implement deployment locks, preflight, migration, service replacement, postflight, observation, and application rollback using release manifests.
|
|
- Implement Redis-loss reconciliation acceptance with backend/worker owners, preserving Asynq `v0.26.0` as a pinned pre-1.0 dependency.
|
|
|
|
Checkpoint: an induced migration/readiness/font/worker failure preserves or restores the prior release.
|
|
|
|
### Phase 4: CI/CD and provenance
|
|
|
|
- Add deployment-repository validation and production-release workflows.
|
|
- Coordinate service-repository workflows that publish immutable images and send candidate events only after required checks.
|
|
- Enforce protected environment approval, serialized release, registry digest retention, SBOM/provenance/vulnerability gates, and secret isolation.
|
|
|
|
Checkpoint: PRs never deploy; backend/frontend candidate events cannot directly mutate production; one manifest controls a tested release and rollback.
|
|
|
|
### Phase 5: observability and recovery
|
|
|
|
- Implement bounded structured container logging, Prometheus/exporters/Grafana, dashboards, and symptom alerts with runbook links.
|
|
- Implement encrypted logical PostgreSQL backup, remote verification, three-local/seven-remote retention, and telemetry.
|
|
- Implement exact-destination guarded restore into an empty destination and application validation.
|
|
- Rehearse restore and record achieved RPO/RTO.
|
|
|
|
Checkpoint: induced failures are diagnosable from telemetry, backup age alert fires, and clean restore succeeds within two hours.
|
|
|
|
### Phase 6: bootstrap and full acceptance
|
|
|
|
- Convert the reviewed first-server checklist into only the minimum approved automation, retaining manual safety/identity gates.
|
|
- Bootstrap the selected host, stage DNS/TLS, deploy a release, run one-time import/credentials, and complete alert/backup/restore handoff.
|
|
- Execute every measurable criterion from the Platform Operations spec and retain non-sensitive evidence.
|
|
|
|
Checkpoint: stakeholder review confirms release, rollback, Redis-loss, backup/restore, exposure, SLO, and operator-boundary evidence.
|
|
|
|
## Risks and mitigations
|
|
|
|
| Risk | Impact | Mitigation |
|
|
|---|---|---|
|
|
| Single-host failure | Full outage | Immutable rebuild path, encrypted off-host PostgreSQL backup, two-hour restore rehearsal, documented DNS/host replacement |
|
|
| Cross-repository race | Incompatible release | Candidate events only, serialized deployment lock, exact manifest, compatibility metadata, one coordinator |
|
|
| Migration blocks rollback | Extended outage/data risk | Expand/contract, measured locks, fresh verified backup, no automatic down migration |
|
|
| Redis loss drops in-flight queue/cache | Partial analytics interruption | AOF for routine restart, PostgreSQL authority, idempotent stale-run reconciliation, cold-cache tests |
|
|
| SSE proxy buffering/timeouts | Dashboard appears stuck | Exact route, buffering/compression/cache off, heartbeat-aware timeout, integration test |
|
|
| Docker privilege compromise | Host root compromise | Approve rootless/dedicated model, restricted machine identity/SSH, no daemon TCP exposure, minimal host users |
|
|
| Secrets leak through Compose/log/backup | Credential/data compromise | File-mounted least privilege, redaction tests, no env in backup, rotate-first incident rule |
|
|
| Backup exists but is unusable | Data loss beyond RTO | Archive/checksum/remote verification plus monthly clean restore |
|
|
| Font missing/divergent | Broken approved typography/license trail | Frontend-owned tracked inputs at `64f5ac2+`, build/static smoke, digest provenance, no second path |
|
|
| Monitoring overload/noise | Lost diagnostics or ignored alerts | Bounded labels/retention, symptom alerts, test-fire, only page/ticket severities |
|
|
|
|
## Parallelization after contracts are fixed
|
|
|
|
- Safe in parallel: Nginx focused tests, backup/restore tooling, observability dashboards, and CI validation, provided the shared manifest/config schemas are merged first.
|
|
- Sequential: schema migrations, production deploy/rollback, database restore, DNS/TLS cutover, and one-time import.
|
|
- Coordinated across repositories: backend commands/health/migration metadata, frontend image/font health, and candidate-event payload. Providers define contracts before consumers implement them.
|
|
|
|
## Completion gate
|
|
|
|
- Every task in `tasks/todo.md` meets acceptance and verification evidence.
|
|
- All spec acceptance criteria pass on a production-shaped host.
|
|
- No unresolved decision remains hidden in a default or script.
|
|
- Active/previous release manifests, SLO dashboards, alerts, backup, restore rehearsal, and operator handoff are complete.
|
|
- Backend/frontend repositories contain only their approved coordinated changes; the deployment repository contains no application source, raw data, secrets, credentials, TLS keys, backups, database volumes, or duplicate font/license files.
|