starcoach-hackathon/zarinpal-analytics-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: specify platform operations

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 18:11:01 +03:30
parent 07c61fdd09
commit d651e0c845
2 changed files with 541 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

521
docs/PLATFORM-OPERATIONS-SPEC.md Normal file
View File

@@ -0,0 +1,521 @@
# Platform Operations Capability Spec
- Status: Draft — ready for stakeholder review
- Implementation authorization: Not granted
- Capability: Platform operations
- Depends on: all deployable backend and frontend capability contracts
- Last updated: 2026-08-20
## Objective
Provide a production-shaped, single-server hackathon platform for `zarinpal.amiirkhl.ir` that is reproducible, private by default, observable, recoverable, and reversible. The deployment repository is the operational control plane; it assembles immutable backend and frontend releases without absorbing either service's Git history or application code.
Success means an operator can bootstrap a clean server, deploy an exact three-repository release, prove the public and internal health contracts, run migrations and the one-time CSV import deliberately, recover PostgreSQL from an encrypted off-host backup, and roll application images back without guessing which revisions were active.
## Scope
### In scope
- Production Docker Compose topology and service boundaries
- Nginx routing, TLS, canonical-host, static asset, API, and SSE behavior
- PostgreSQL, Redis, backend API, frontend, and analytics worker lifecycle
- Liveness, readiness, startup ordering, graceful shutdown, and resource bounds
- Dedicated migration, one-time CSV import, and demo-credential administrative jobs
- Environment, configuration, secret, volume, network, and host exposure policy
- Cross-repository CI quality gates, immutable image publication, release orchestration, and rollback
- Structured logs, bounded local retention, metrics, dashboards, alerts, and release evidence
- Encrypted PostgreSQL backup, off-host retention, guarded restore, and restore rehearsal
- Safe first-server bootstrap and operator boundaries
- Ordinary frontend build consumption of the approved licensed font assets
### Out of scope
- Backend or frontend application implementation
- Owning ZarinPal's future production payment-write path
- Scheduled or watched CSV ingestion
- Kubernetes, multi-region, automatic failover, or zero-downtime database migration
- Redis backup or treating Redis as a source of truth
- Public administration, database, Redis, metrics, or worker endpoints
- ZarinPal SSO implementation
- Creating Compose, Nginx, Dockerfile, script, workflow, or environment artifacts before this spec is approved
## Assumptions
1. Production is one Linux server with sufficient CPU, memory, and SSD capacity; Compose is the accepted hackathon orchestrator.
2. DNS for `zarinpal.amiirkhl.ir` can point directly to the server and ports 80/443 can accept ACME HTTP-01 and application traffic.
3. The Git forge is Gitea-compatible and provides protected branches, Actions or an equivalent trusted runner, a private OCI registry, deployment environments, secrets, and an authenticated cross-repository event mechanism. Exact forge and runner versions remain a recorded decision.
4. Backend images contain independently runnable API, worker, migration, importer, and administrative commands from one reviewed backend revision. The importer is not the API entrypoint.
5. The frontend is a static/client-rendered SvelteKit build served by its own private web container. Nginx is the only public reverse proxy.
6. PostgreSQL 18 and Redis 7 are the Qlockify-aligned initial major versions; final image tags are pinned to patch versions and immutable digests in a release manifest after compatibility tests.
7. A short maintenance window during a single-host application rollout is acceptable. The target is no more than 120 seconds of failed readiness, not a claim of zero downtime.
8. Backup RPO is 24 hours and restore RTO is two hours for the hackathon platform.
9. The approved `IRANYekanXVFaNumVF.woff2` and required `FontLicense.txt` are frontend-owned, tracked under `static/fonts` by frontend commit `64f5ac2`, and legally distributable for this project/domain. Platform Operations neither copies nor privately injects them.
Any invalid assumption changes this spec before implementation.
## Architecture and trust boundaries
```text
Internet
|
| TCP 80/443 only
v
+---------------- edge network ----------------+
| nginx (TLS, routing, limits, access logs) |
+--------------------+--------------------------+
|
+-----------+-----------+
| private app network |
v v
frontend static web backend API
|
+------------+------------+
| private data network |
v v
PostgreSQL <--- worker ---> Redis
^ ^
| |
one-off migrate/import/admin jobs
Private operator path (SSH tunnel/VPN only)
-> Prometheus/Grafana and host diagnostics
```
Trust boundaries are the internet-to-Nginx edge, Nginx-to-application network, application-to-data network, CI-to-private-registry boundary, deployment-runner-to-server SSH boundary, operator-to-administrative-job boundary, and backup-host-to-object-storage boundary. Assets worth protecting include merchant payment analytics, pseudonymous payer-card keys, credentials/session digests, database contents, CI/deploy keys, backup encryption keys, TLS private keys, and release provenance.
## Compose service topology
| Service | Image/process boundary | Networks | Persistent state | Public exposure | Restart/lifecycle |
|---|---|---|---|---|---|
| `nginx` | Pinned Nginx image with reviewed configuration | `edge`, `app` | ACME webroot and certificate volume only | `80:80`, `443:443` | `unless-stopped`; starts only after frontend/backend readiness for production traffic |
| `certbot` | Pinned ACME client | `edge` | ACME account/certificate volume | None | One-shot initial issuance plus bounded renewal loop or host timer; not a health dependency after a valid certificate exists |
| `frontend` | Immutable frontend image containing static build output | `app` | None | `expose` only | `unless-stopped`; health checks a local static response |
| `backend` | Immutable backend image, API command | `app`, `data` | None | `expose` only | `unless-stopped`; waits for successful migration and healthy data dependencies |
| `worker` | Same backend digest, worker command; Asynq `v0.26.0` pinned | `data` | None | None | `unless-stopped`; event-driven only, never scheduled |
| `postgres` | PostgreSQL 18 patch/digest | `data` | `postgres_data` | None | `unless-stopped`; graceful stop with a bounded timeout |
| `redis` | Redis 7 patch/digest with authentication and AOF `everysec` | `data` | `redis_data`, excluded from backup | None | `unless-stopped`; routine restart persistence only, still disposable for disaster recovery |
| `migrate` | Same backend digest, migration command | `data` | None | None | One-off; must complete successfully before backend/worker replacement |
| `import` | Same backend digest, explicit importer command/profile | `data` | Read-only operator-selected input mount | None | One-off only; never part of normal `up`, startup, cron, or health dependency |
| `admin` | Same backend digest, explicit credential/maintenance command/profile | `data` | Explicit output mount when requested | None | One-off only; output destination and permissions are operator-selected |
| `prometheus` | Pinned Prometheus image | `monitoring`, scrape access to private exporters | `prometheus_data` with seven-day/size retention | Loopback/operator path only | `unless-stopped`; platform remains functional if monitoring is down |
| `grafana` | Pinned Grafana image | `monitoring` | `grafana_data` | Loopback/operator path only | `unless-stopped`; no Nginx public route by default |
| exporters | Pinned node/container/PostgreSQL/Redis exporters as approved | `monitoring`, minimum required target network | None | None | Least-privilege, read-only metrics only |
The implementation uses the current Compose Specification, not a legacy numeric schema version. Every image uses an explicit patch tag plus digest. Floating `latest`, floating major-only application images, server-side source builds, and mutable branch references are prohibited in production.
### Network and exposure rules
- Only Nginx publishes host ports 80 and 443.
- PostgreSQL, Redis, backend, worker, frontend, exporters, Prometheus, and Grafana do not publish internet-facing ports.
- `edge`, `app`, `data`, and `monitoring` networks are separate. A service joins only the networks it needs.
- The frontend cannot reach PostgreSQL or Redis. Nginx cannot reach PostgreSQL or Redis. Exporters receive only the minimum read-only connectivity needed.
- Operator UIs bind to `127.0.0.1` when a host port is unavoidable and are reached through SSH tunneling or an approved private network.
- Host firewall default is deny inbound; allow established traffic, SSH from an operator allowlist, and TCP 80/443. Docker's firewall behavior must be verified from an external host, not inferred from host rules alone.
### Volumes and filesystem policy
| Volume/input | Purpose | Backup | Required handling |
|---|---|---|---|
| `postgres_data` | Authoritative database cluster | Logical PostgreSQL backup, not raw live-volume copying | Dedicated named volume; capacity and inode alerts |
| `redis_data` | AOF for routine queue/cache restart continuity | No | Rebuildable; loss triggers idempotent run reconciliation and cold cache |
| `letsencrypt_data` | ACME account and certificate material | No database backup; separately recover/reissue | Read-write only to certbot, read-only to Nginx |
| `certbot_webroot` | HTTP-01 challenge | No | No application files |
| `prometheus_data` | Short-retention metrics | No | Seven-day and size-bounded retention |
| `grafana_data` | Local dashboard state | Optional export of declarative dashboards only | Dashboards/alerts should be tracked as config after approval |
| CSV input bind | One-time archive input | Source archive retained by operator policy, outside Git | Absolute validated host path, read-only mount, checksum before import |
| admin output bind | One-time credential output if file selected | No | Absolute validated path outside repository; restrictive mode; operator removes/escrows it |
Containers use read-only root filesystems where practical, drop Linux capabilities, set `no-new-privileges`, run as non-root, and use bounded tmpfs mounts for writable runtime paths. CPU/memory/PID limits and log rotation are mandatory. Exact limits are established by load tests, with safe non-zero reservations left for PostgreSQL and the host.
## Startup, migration, import, and shutdown
### Normal release startup
1. Validate host identity, intended environment, clean deployment control-plane checkout, release-manifest signature/checksum, required secret files, free disk, current backup age, and Compose-rendered configuration.
2. Pull images by digest. Do not build from server checkouts.
3. Start/verify PostgreSQL and Redis health.
4. Acquire a deployment lock and run the `migrate` one-off service using the candidate backend digest.
5. Require migration exit code zero and record migration version/duration. Application startup never runs migrations or automatic schema creation.
6. Replace backend and worker with the candidate digest. Wait for backend and worker readiness.
7. Replace frontend, then validate its local health and the font asset response from the candidate image.
8. Validate Nginx configuration, reload/recreate Nginx, and run same-origin public smoke checks.
9. Observe the release for 15 minutes. Failure thresholds trigger rollback to the previous release manifest.
10. Record the active release manifest, actor, timestamps, health evidence, and outcome.
Compose dependency conditions are a startup aid, not the deployment controller. `service_healthy` gates dependencies and `service_completed_successfully` gates migrations, but the release workflow still performs explicit pre/post checks.
### Migration contract
- Migrations are reviewed, versioned backend artifacts run exactly once per release under an advisory/exclusive migration lock.
- A migration must be backward-compatible with the currently running application for the rollback window: expand first, deploy readers/writers, clean up only in a later approved release.
- Destructive, long-locking, or non-reversible migrations require an explicit operator approval, fresh verified backup, measured lock/duration evidence on production-shaped data, and a dedicated rollback/recovery plan.
- Rollback never automatically runs a down migration. The default rollback restores prior application images against the expanded compatible schema.
- A failed migration stops the release before application replacement. It must not be retried blindly when partial effects are possible.
### One-time CSV import contract
- Import is an explicit operator action after schema migration, never a Compose startup command, scheduled service, file watcher, or deployment trigger.
- The operator supplies an absolute input path outside Git. The deployment wrapper resolves and validates that path, mounts exactly the file read-only, and records its SHA-256 through the backend import manifest.
- Import runs with one concurrency slot and an import lock. It has bounded database resources and cannot overlap schema migration, restore, or another import.
- Re-running an identical completed checksum/schema version is a safe no-op or explicit verified reconciliation according to the approved Data Foundation spec.
- Failure leaves an incomplete import run with evidence; it never marks data complete. Resume/retry behavior is owned by the backend import contract.
- Demo credential generation is a separate `admin` job after a successful import. Plaintext output never enters Compose logs, CI logs, Git, or the release manifest.
### Shutdown
- Nginx stops accepting new traffic before backend termination; backend and worker receive a termination signal and a bounded drain period.
- Worker shutdown stops fetching new Asynq jobs and allows an active job to finish only within its existing 60-second attempt timeout. Unfinished retryable work remains/reappears through queue semantics.
- PostgreSQL stops last. Forced kill is a failure condition recorded by the deploy workflow.
## Health and readiness contracts
Health responses are generic and contain no versions, dependency addresses, credentials, stack traces, tenant information, or database details.
| Component | Liveness | Readiness | Failure meaning |
|---|---|---|---|
| Nginx | Local config/process endpoint answers | TLS virtual host can serve frontend and proxy backend readiness | Restart for failed liveness; remove public readiness on upstream failure |
| Frontend | Local static sentinel returns 200 | `index.html`, hashed assets, and required font file are readable | Candidate image/build is incomplete |
| Backend | Process/event loop answers without external dependency calls | PostgreSQL reachable, expected migration present, Redis reachable for analytics/session rate limits, and startup validation complete | Do not route new API traffic; do not restart solely for dependency outage |
| Worker | Worker process answers local probe | PostgreSQL/Redis reachable, queue consumer registered, configured concurrency valid, heartbeat fresh | Stop accepting/claiming jobs until ready |
| PostgreSQL | Container process | `pg_isready` against the configured database/user plus application-level migration check from backend | Dependency unavailable or schema mismatch |
| Redis | Process responds to authenticated `PING` | Required queue/cache commands usable and memory policy validated | Analytics/cache coordination unavailable |
Defaults after implementation:
- Health interval: 10 seconds; timeout: 3 seconds; start period: 30 seconds; five consecutive failures before unhealthy, adjusted only from observed cold-start data.
- Public `/health/live` and `/health/ready` return only generic status. Deep dependency details remain in private metrics/logs.
- Liveness never depends on PostgreSQL, Redis, or an external service; dependency outages must not create restart storms.
- Readiness fails closed for a schema mismatch and for dependencies required to serve the routed capability.
## Nginx, domain, and TLS contract
### Routing
- Port 80 serves `/.well-known/acme-challenge/` and redirects all other requests to `https://zarinpal.amiirkhl.ir$request_uri` with a permanent redirect.
- Any alternate hostname redirects to the canonical hostname. Unknown hosts are rejected by a default server and never proxied.
- `/api/` proxies to the backend over the private `app` network with the original host, verified client address chain, scheme, and a generated/preserved request ID.
- The exact authenticated analytics SSE path is generated from the approved API contract and receives a dedicated location block; it is never matched only by a broad accidental prefix.
- `/` proxies to the private static frontend container. The frontend container owns SPA fallback behavior so `/api/` can never fall through to `index.html`.
- Fingerprinted static assets use a one-year immutable cache. HTML and mutable manifests use `no-cache`/revalidation. Font MIME type, `Content-Type`, and same-origin caching are verified.
- Public API documentation, metrics, debug, profiling, database administration, Redis, worker, and Grafana routes are absent unless a later ADR explicitly exposes an authenticated operator path.
### SSE route
- Use HTTP/1.1 upstream, clear hop-by-hop `Connection`, preserve `Cookie`, `Last-Event-ID`, request ID, host, scheme, and client-forwarding headers.
- Disable proxy buffering, proxy cache, response compression, and transformations for this route; send `X-Accel-Buffering: no` as defense in depth.
- Set read timeout safely above the 15-second heartbeat (initial target 75 seconds) and a bounded connect/send timeout. Do not inherit a one-hour general API timeout.
- Do not cache authenticated SSE responses. Reauthorize every connect/replay request; Nginx does not bypass backend session checks.
- Enforce connection/body/header limits without buffering the event stream. Backend user/merchant SSE concurrency limits remain authoritative.
### TLS and HTTP security
- TLS 1.2 and 1.3 only, modern cipher defaults, session reuse, and automated certificate renewal.
- ACME production issuance occurs only after DNS and staging issuance checks pass. Nginx may use a temporary self-signed certificate only during isolated bootstrap; it must not be reported as production-ready.
- Certificate expiry warning at 21 days and page at seven days. Renewal is testable without waiting for expiry.
- Enable HSTS only after a valid renewable certificate is proven; initial target `max-age=31536000; includeSubDomains` only if subdomain ownership is confirmed. Preload requires separate approval.
- Apply `X-Content-Type-Options: nosniff`, a restrictive frame policy, referrer policy, permissions policy, and the frontend-approved CSP. Security headers are tested on HTML, API, error, and redirect responses.
- Login and API request-body limits are small and explicit. The CSV archive is never uploaded through public Nginx.
## Configuration and secret handling
### Configuration classes
| Class | Examples | Storage/delivery |
|---|---|---|
| Public tracked defaults | Domain placeholder, port names, timeouts, concurrency defaults, metric retention | `.env.example`/documented Compose defaults after approval; no real endpoints or credentials |
| Environment-specific non-secret config | Production domain, image digests, resource limits, alert destinations by alias | Signed/private release manifest or protected forge variables |
| Runtime secrets | PostgreSQL/Redis passwords, session/CSRF keys, backup credentials/key, deploy key, Grafana admin credential | Forge/host secret store; mounted as read-only files where supported |
| TLS private material | ACME account and private keys | Restricted certificate volume; never environment variables or Git |
| Licensed frontend font inputs | `static/fonts/IRANYekanXVFaNumVF.woff2`, `static/fonts/FontLicense.txt` | Tracked and consumed in the frontend repository's ordinary build context; not copied into deployment Git |
Requirements:
- Production secret files live outside the checkout under a root/operator-controlled directory such as `/etc/zarinpal-analytics/secrets`, mode 0600 or stricter, and are mounted only into consumers.
- Compose interpolation files contain non-secret release/config values where possible. Secrets are not passed on command lines, embedded in image layers, printed by `compose config`, or included in diagnostic bundles.
- A committed example names every required variable and supplies safe placeholders/validation rules, never plausible credentials.
- Backend and worker share only the secrets each command needs. Frontend receives no runtime secret.
- CI test credentials are isolated from production. The deploy job gets no database or backup decryption key.
- Secret rotation is documented and can be performed without rebuilding unrelated images. If a secret enters Git or logs, revoke/rotate first, then purge as a separate incident task.
### Font build dependency
The approved decision, implemented in frontend commit `64f5ac2`, places `IRANYekanXVFaNumVF.woff2` and `FontLicense.txt` in the frontend repository under `static/fonts`. Platform Operations creates no separate injection workflow or deployment-repository copy. Frontend CI verifies both required tracked files exist, the production build emits the expected font resource, and the candidate container serves it with the correct MIME type. Deployment records only the frontend Git SHA and immutable image digest; the font travels as part of that ordinary frontend artifact.
## Cross-repository CI/CD and release provenance
### Service repository triggers
**Backend pull request:** format/lint, unit/integration tests, race tests, `go vet`, `golangci-lint`, `govulncheck`, migration verification, importer fixtures, queue/retry/recovery tests against pinned pre-1.0 Asynq `v0.26.0`, image build, image scan, and SBOM/provenance generation. No production deploy.
**Backend protected-main/tag:** after all gates pass, build once, publish the immutable backend image to the private registry, capture its digest plus backend SHA/migration compatibility metadata, and send an authenticated candidate event to the deployment repository.
**Frontend pull request:** frozen dependency install, format/lint, `svelte-check`, unit/component tests, Playwright critical flows, accessibility/RTL/reduced-motion checks, production build, required font/license file presence, image build, scan, and SBOM/provenance generation. No production deploy.
**Frontend protected-main/tag:** after all gates pass, build once, publish an immutable frontend image, capture digest plus frontend SHA/API compatibility metadata, and send an authenticated candidate event to the deployment repository.
**Deployment pull request:** documentation checks plus, after implementation is approved, Compose render/validation, Nginx syntax tests, shell lint/tests, secret scanning, policy checks for published ports/digests/health checks, backup/restore tests, and integration smoke using explicitly selected backend/frontend candidate digests.
**Deployment protected-main/tag/manual release:** assemble or approve a release manifest, acquire a single production deployment lock, run preflight/migration/deploy/postflight, and monitor. A service event creates a candidate; it does not independently SSH to production or mutate the server.
### Release manifest
Every production release records:
- release id and creation timestamp
- deployment, backend, and frontend Git SHAs
- immutable image names and digests for every service
- PostgreSQL/Redis/Nginx/monitoring image digests
- expected migration version and compatibility direction
- Compose/config schema revision and non-secret config checksum
- SBOM/provenance references and vulnerability-gate result
- actor/approval, preflight backup id, deploy outcome, and previous release id
The manifest contains no secret values, license identifier, raw font copy, credentials, CSV path/content, or database data. It is stored in the deployment release history and on the server as an append-only operational record. Production deploys consume digests from this manifest and never `git reset --hard` nested service checkouts or rebuild source on the server.
### Release gate and rollback
- Only protected immutable revisions can become candidates. Required checks and one human production approval are mandatory.
- Deployments are serialized; overlapping backend/frontend/deployment events coalesce into a newly reviewed manifest rather than racing.
- Pre-deploy requires a successful backup within 24 hours; a destructive migration requires a fresh on-demand verified backup.
- Post-deploy gates: Compose health, TLS/canonical redirect, static/font response, login page, generic liveness/readiness, authenticated smoke with a dedicated non-sensitive account, one analytics start/SSE heartbeat, worker/queue health, and no threshold breach during the 15-minute watch.
- Application/config failure rolls back to the previous manifest's image digests and compatible config. The failed candidate remains recorded.
- Database down-migration is never automatic. If a migration is not backward-compatible, release approval requires a separate database recovery plan and extended outage acknowledgment.
- Rollback completes within 15 minutes for an application-only release under normal registry/network conditions.
## Observability and reliability objectives
### Operator questions
Telemetry must answer:
1. Can users reach, authenticate, and load the dashboard through the canonical TLS endpoint?
2. Which release and request/run/section caused a failure, and was PostgreSQL, Redis, queueing, calculation, or SSE delivery responsible?
3. Are queue delay, calculation duration, database saturation, Redis memory, disk, certificate age, or backup age approaching a user-visible limit?
4. Can the current backup be restored to a clean destination within the stated RTO?
### Logs
- Application and worker logs are structured JSON to stdout with stable event names, UTC timestamp, level, service, release id, request id, analytics run/section/job id where applicable, safe failure class, and duration.
- Nginx access logs are structured and include request id, route template/category, status, bytes, request/upstream duration, and upstream status. They exclude cookies, authorization headers, query strings that may contain sensitive values, and response bodies.
- Never log credentials, raw session tokens/cookies, password hashes, payer-card keys, raw CSV rows, database URLs, Redis URLs/passwords, backup keys, or full merchant payloads.
- Docker log storage uses a bounded driver/rotation policy. Initial target is 10 MiB per file, three files per container; validate against incident needs and disk budget.
- Production debug logging is off by default and time-bounded when enabled. Logs retain seven local days or less under a size cap; longer retention requires an approved privacy/cost decision.
### Metrics and dashboards
- HTTP RED metrics by bounded route template/method/status class; latency histograms expose p50/p95/p99.
- Backend dependency RED metrics for PostgreSQL and Redis without query text, merchant id, request id, raw URL, or error text as labels.
- Worker: queue depth/oldest age, active jobs, global/per-merchant saturation as bounded aggregate series, section duration, retry/failure class, cache hit/miss, SSE connections/reconnect/replay/snapshot fallback, and suppression rate.
- PostgreSQL: availability, connections/pool saturation, transaction/lock pressure, slow-query aggregate, database/volume size, replication metric only if replication exists.
- Redis: availability, memory/limit, eviction, rejected connection, persistence status, queue depth, and key expiry aggregate.
- Host/container: CPU, memory, load, filesystem space/inodes, OOM/restarts, network, and certificate days remaining.
- Backup: last success timestamp, duration, size, remote upload/checksum status, retention count, and last successful restore rehearsal.
- Metric labels are bounded; merchant/user/card/request identifiers never become metric labels.
Prometheus and Grafana remain private. Dashboards show service overview, release health, analytics worker/SSE, PostgreSQL/Redis capacity, host capacity, and backup/TLS status. Alerts link to a runbook and use only `page` or `ticket` severity.
### Initial SLOs and alert thresholds
| Objective | Target / measurement | Initial alert |
|---|---|---|
| Public availability | 99.5% successful synthetic HTTPS readiness over rolling 7 days, excluding approved maintenance | Page after 5 minutes of consecutive failure |
| Non-streaming API errors | <1% unexpected 5xx over 5 minutes | Page at >2% for 5 minutes with minimum traffic |
| Non-analytics API latency | p95 <500 ms over 5 minutes under acceptance load | Ticket at p95 >500 ms for 15 minutes; page at >2 s for 5 minutes |
| Analytics start response | p95 <1 second, work queued/deduplicated before response | Page at p95 >3 seconds or >2% 5xx for 5 minutes |
| SSE delivery | Connection established <2 seconds p95; heartbeat gap <=20 seconds while healthy | Page when synthetic stream misses two heartbeats |
| Section jobs | Queue delay p95 <5 seconds; attempts always <=60 seconds; no more than two transient retries | Ticket if oldest queue age >30 seconds; page >2 minutes |
| Capacity | Host/data volumes keep >=20% and >=10 GiB free; no sustained DB/Redis saturation | Ticket below 25%; page below 15% or projected exhaustion <24 hours |
| Backup | Encrypted off-host backup succeeds every 24 hours | Page when last verified remote backup age >30 hours |
| Restore | Monthly clean-destination rehearsal completes within two hours | Ticket immediately on rehearsal failure |
| TLS | Valid certificate and automated renewal | Ticket <21 days; page <7 days |
Targets are hackathon baselines, not evidence of current performance. Load and failure tests must validate or revise them before approval for production use.
## Backup, restore, and disaster recovery
### Backup contract
- PostgreSQL is the only required application data backup. Redis, Prometheus, Grafana runtime state, TLS material, environment files, secrets, source CSV, and licensed font files are excluded.
- Run a daily PostgreSQL logical backup using the PostgreSQL 18 client matching the server major. Use custom/directory format suitable for `pg_restore`, include globals/roles in a separately controlled form if required, and fail on command errors.
- A backup package contains the database archive, checksums, PostgreSQL/server version, schema migration version, release id, start/end timestamps, size/count evidence, and a format version. It contains no environment files or secret values.
- Encrypt before durable local or remote storage with an authenticated, approved scheme. The decryption key is stored separately from the server and object-store credentials.
- Upload to private S3-compatible object storage over TLS. Enable provider-side versioning/object lock when available.
- Initial retention aligns with Qlockify operational practice: latest three encrypted local backups and latest seven encrypted remote backups. Retention deletion runs only after successful upload and checksum verification.
- A successful dump alone is not a successful backup: archive listing/checksum, remote object verification, and manifest completion are required.
### Restore contract
- Restore defaults to a new empty database on a non-production destination. PostgreSQL documents that dumps can execute source-controlled SQL; only trusted project backups may be restored.
- Before extraction/decryption, validate file type, checksum, manifest schema, expected object key, available space, PostgreSQL compatibility, and exact resolved destination.
- Production restore requires all of: explicit environment flag, exact canonical hostname, exact database name, typed one-time confirmation phrase containing the backup id, maintenance mode, active deployment lock, current-state backup when feasible, and an operator audit record.
- Never restore over a running production database in place. Stop writers, terminate sessions deliberately, restore into a newly created empty database/cluster, validate, then switch connectivity according to the approved implementation runbook.
- Use `pg_restore` with error-stop semantics and appropriate ownership handling. Run migrations only if the selected release contract requires them; never silently migrate beyond the chosen restore point.
- Post-restore validation checks database/schema/import manifests, representative counts, tenant isolation smoke, login/session behavior, analytics cold-cache recomputation, and Redis-empty recovery before reopening traffic.
- Rehearse restore monthly and before a destructive migration. Record RPO/RTO achieved, backup id, release id, validation results, and remediation.
### Disaster scenarios
| Failure | Expected response |
|---|---|
| Redis loss/corruption | Stop worker if unsafe, replace empty Redis, reconcile stale analytics runs idempotently, recompute cache; no PostgreSQL restore |
| Frontend/backend image failure | Roll back to previous release manifest digests; no database restore |
| Bad backward-compatible migration | Roll back app image, retain expanded schema, ship corrective migration |
| PostgreSQL logical corruption/operator error | Enter maintenance, select trusted backup, guarded restore to empty destination, validate, then reopen |
| Host loss | Bootstrap replacement host, restore secrets through separate escrow, deploy chosen manifest, restore PostgreSQL, issue/restore TLS, run validation |
| Object-store outage | Preserve encrypted local retention, alert, retry bounded upload; do not delete the last verified local backup |
| Certificate renewal failure | Alert at 21 days, correct DNS/ACME path, test renewal; never weaken TLS to hide expiry |
## Security model and failure modes
### STRIDE summary
| Threat | Platform control |
|---|---|
| Spoofed deploy/operator | Protected environments, hardware-backed/MFA human accounts where available, pinned SSH host key, least-privilege machine identity, signed/checksummed release manifest |
| Tampered image/config/backup | Digest-pinned images, protected release history, checksums, private registry, encrypted authenticated backups, trusted-source restore only |
| Repudiated production action | Append-only release/restore records with actor, approval, exact revisions, timestamps, and outcome |
| Information disclosure | Private networks, minimal ports, file-mounted secrets, log allowlists, no env in backups, no public monitoring/admin routes |
| Denial of service | Nginx limits/timeouts, bounded pools/resources/PIDs/logs, application rate limits, queue concurrency and 60-second job timeout |
| Elevation of privilege | Non-root containers, dropped capabilities, `no-new-privileges`, separated networks/secrets, restricted deploy command and operator path |
### Failure behavior
- Missing/malformed configuration or secret: fail validation before mutation and name only the missing key/path, never its value.
- Unhealthy PostgreSQL/Redis: do not start/route dependent candidate services; preserve the last healthy release.
- Migration failure: stop release; never start candidate API/worker or auto-run down migration.
- Import failure: isolate the incomplete import record; normal deploy remains independent.
- Frontend/font validation failure: reject candidate before public switch; serve previous frontend.
- Backend readiness failure: keep/restore previous backend and record candidate logs/health evidence.
- Worker unavailable: API reports typed section degradation; queue age alert fires; ready cached sections remain usable.
- Redis memory exhaustion: reject unsafe new work, alert, and preserve PostgreSQL; do not silently evict authoritative state because Redis has none.
- Monitoring outage: application continues, but release gate warns/fails if the 15-minute observation cannot be performed.
- Backup/upload failure: retain previous verified backups, alert, and block destructive migrations.
- Restore validation mismatch: abort before destination mutation.
- Certificate missing/expired: production readiness fails; do not expose plaintext application traffic.
- Registry/forge unavailable: current release continues; deploy does not fall back to local mutable builds.
## Always / Ask / Never boundaries
### Always
- Deploy immutable, scanned image digests recorded with all three repository SHAs.
- Validate exact environment, target path, configuration, backup age, health, migration, and rollback manifest before mutation.
- Keep Nginx as the only public entry point and use TLS for all external traffic.
- Separate migrations, CSV import, credential generation, API startup, and worker startup.
- Keep Asynq pinned to `v0.26.0` and explicitly treat it as pre-1.0; test queue/retry/recovery before any upgrade.
- Mount secrets with least privilege, keep them out of Git/logs/backups/images, and rotate after exposure.
- Bound resources, timeouts, retries, logs, metrics cardinality, backup retention, and operator actions.
- Back up PostgreSQL encrypted off-host, verify the artifact, and rehearse restores.
- Preserve the current healthy release until candidate readiness and smoke checks pass.
- Consume approved font/license files only through the ordinary frontend build and immutable frontend artifact.
### Ask first
- Expose any new port, hostname, admin/debug/docs/metrics route, or third-party telemetry endpoint.
- Change PostgreSQL/Redis major version, persistence policy, network boundary, SLO, RPO/RTO, retention, or resource limit materially.
- Run a destructive/non-backward-compatible migration, production restore, import replacement, or secret rotation that causes downtime.
- Change deploy approval/trigger behavior, registry/runner trust, backup provider/encryption scheme, TLS/HSTS scope, or SSH access model.
- Add a scheduled service, new external integration, new persistent volume, or public CDN.
- Change frontend ownership/location of the licensed font/license inputs.
### Never
- Push from an agent session, force-reset user history, or edit backend/frontend repositories from this deployment task.
- Build production images from mutable server checkouts or deploy branch names/floating tags.
- Publish PostgreSQL, Redis, backend, worker, monitoring, or Docker daemon ports to the internet.
- Run migrations or CSV import inside API startup; schedule/watch/poll the challenge CSV.
- Back up or restore Redis as authoritative data, or include `.env`, secret files, TLS private keys, raw CSV, credentials, or font copies in database backups.
- Log secrets, raw session/cookie values, payer-card keys, raw payment rows, database/Redis URLs, or high-cardinality identifiers as metric labels.
- Automatically run down migrations or overwrite production during restore without the complete confirmation guard.
- Add private font injection, copy the font/license into deployment Git, or diverge from the approved ordinary frontend build input.
- Disable failed health/security/CI gates merely to complete a release.
## Measurable acceptance criteria
### Topology and exposure
- `docker compose config` validates and every production image resolves to an immutable digest.
- An external port scan finds only approved SSH plus TCP 80/443; PostgreSQL, Redis, backend, worker, frontend, Prometheus, and Grafana are unreachable directly.
- Network inspection proves frontend/Nginx cannot connect to PostgreSQL/Redis and only required paths exist.
- Container policy tests prove non-root/read-only/capability/resource/log constraints or document an approved exception.
### Routing and TLS
- HTTP and alternate hosts redirect to exactly `https://zarinpal.amiirkhl.ir`; unknown hosts are rejected.
- TLS accepts only 1.2/1.3 with a valid renewable certificate; security headers pass documented checks.
- `/api/` reaches only backend; application routes reach frontend; no SPA fallback captures API errors.
- The authenticated SSE stream delivers a ready event without buffering, emits heartbeats within 20 seconds, preserves replay ids, and survives a controlled Nginx reload/reconnect.
### Lifecycle and data safety
- A clean deployment waits for PostgreSQL/Redis health, runs migration once, then starts ready API/worker/frontend services.
- A migration failure prevents candidate API/worker startup and leaves the prior release recoverable.
- Normal startup/deploy never runs the importer. An explicit identical import re-run is idempotent and produces no duplicate business rows.
- Redis deletion followed by restart produces safe analytics recomputation and no loss of PostgreSQL facts or sessions.
- Graceful deployment stops new work and leaves no section attempt running beyond its 60-second bound.
### CI/CD and rollback
- Backend/frontend PRs cannot deploy; protected revisions publish one immutable image each only after their documented gates.
- Candidate events cannot mutate production directly; one serialized deployment workflow assembles all three revisions into a release manifest.
- A deployed instance exposes/report its release id internally, and it matches the manifest's three SHAs/digests.
- An induced candidate readiness failure preserves or restores the previous release. An application-only rollback completes within 15 minutes.
- No production job receives CI test credentials, and deployment artifacts/logs contain no secrets.
### Font ownership
- Frontend CI and deployment smoke verify `/fonts/IRANYekanXVFaNumVF.woff2` (or the exact frontend-approved emitted URL) returns 200 with the correct font content type.
- The deployment repository contains neither font binary nor `FontLicense.txt`; the active frontend image digest maps to the frontend SHA that owns both files.
- No private asset fetch/injection step exists in deployment CI/CD.
### Observability and recovery
- An induced backend 5xx is findable by request id and release id using structured logs without exposing sensitive fields.
- RED/USE, worker/SSE, database/Redis, host, backup, and TLS metrics appear with bounded labels and the defined alerts are test-fired.
- A daily encrypted backup is uploaded, checksum-verified, retained as three local/seven remote artifacts, and no excluded secret/config/runtime asset appears in it.
- A clean-destination restore of a selected backup completes within two hours, passes application/tenant/analytics validation, and records achieved RPO/RTO.
- A production restore attempt without any required confirmation factor aborts before database mutation.
## Implementation verification commands (to be finalized)
These command interfaces are specification targets, not implemented commands:
```text
Validate: docker compose --env-file <non-secret-release-env> config --quiet
Inspect: docker compose --env-file <non-secret-release-env> ps
Logs: docker compose --env-file <non-secret-release-env> logs --since <duration> <service>
Migrate: deployment command -> one-off migrate service at candidate backend digest
Import: deployment command -> one-off import service with exact read-only source path
Backup: deployment command -> encrypted PostgreSQL backup plus remote verification
Restore: deployment command -> preflight, guarded empty-destination restore, validation
Rollback: deployment command -> previous immutable release manifest
```
Exact script names and flags are implementation decisions constrained by the operator safety contract above. They are not authorized by this document's draft status.
## Remaining decisions
1. Confirm server OS/provider, CPU/RAM/disk, backup object-store provider, and whether object lock/versioning is available.
2. Confirm Gitea/Actions/runner/registry versions and choose the supported authenticated cross-repository candidate event mechanism.
3. Decide rootless Docker versus a dedicated rootful Engine deploy account; document the chosen low-port and privilege model.
4. Approve exact PostgreSQL 18, Redis 7, Nginx, certbot, Prometheus/Grafana/exporter patch versions and digests after compatibility/security checks.
5. Approve concrete CPU/memory/PID/pool/log/metrics-retention budgets after production-shaped load testing.
6. Approve the exact backend health endpoints, worker heartbeat contract, API/SSE paths, OpenAPI compatibility metadata, and dedicated smoke-test account.
7. Choose the encrypted backup format/key-management mechanism and production restore database-switch strategy.
8. Decide whether certificate renewal runs as a Compose service or a host systemd timer; both must preserve Nginx reload and private-key permissions.
9. Confirm whether one human production approval is available in the forge and who owns deploy, backup, restore, and incident roles during the hackathon.
10. Confirm the licensed font terms permit the approved direct frontend-repository tracking and built-image distribution for `zarinpal.amiirkhl.ir`; application of that approved repository change remains frontend-owned and outside this task.
## References
- Docker Compose application model: <https://docs.docker.com/compose/intro/compose-application-model/>
- Docker Compose startup order and health conditions: <https://docs.docker.com/compose/how-tos/startup-order/>
- Docker logging driver configuration: <https://docs.docker.com/engine/logging/configure/>
- Docker rootless mode: <https://docs.docker.com/engine/security/rootless/>
- Nginx proxy module, including buffering/timeouts: <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>
- Nginx TLS module: <https://nginx.org/en/docs/http/ngx_http_ssl_module.html>
- PostgreSQL 18 backup and restore: <https://www.postgresql.org/docs/current/backup.html>
- PostgreSQL SQL dump/restore guidance: <https://www.postgresql.org/docs/current/backup-dump.html>
- Gitea Actions overview: <https://docs.gitea.com/usage/actions/overview/>
- Gitea runner documentation: <https://docs.gitea.com/runner/>
- Local read-only reference: `D:/Programing/Scripts/Qlockify/qlockify-deployment`