|
|
|
@@ -1,519 +1,508 @@
|
|
|
|
# Platform Operations Capability Spec
|
|
|
|
# Platform Operations Capability Spec
|
|
|
|
|
|
|
|
|
|
|
|
- Status: Draft — ready for stakeholder review
|
|
|
|
- Status: Draft revision — confirmed intent, ready for detailed stakeholder review
|
|
|
|
- Implementation authorization: Not granted
|
|
|
|
- Implementation authorization: Not granted
|
|
|
|
- Capability: Platform operations
|
|
|
|
- Capability: Platform operations
|
|
|
|
- Depends on: all deployable backend and frontend capability contracts
|
|
|
|
- Depends on: all deployable backend and frontend capability contracts
|
|
|
|
- Last updated: 2026-08-20
|
|
|
|
- Last updated: 2026-08-20
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Confirmed intent
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Deploy `zarinpal.amiirkhl.ir` on one Ubuntu server using the proven Qlockify operating model: three nested Git repositories, Docker Compose builds performed on the server, Nginx as the only public entry point, automatic Let's Encrypt TLS, PostgreSQL 18, Redis 7 Alpine, a Go API and event-driven worker, and a static frontend.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Production deployment is manual. The operator pulls the latest `main` from all three repositories, records the resulting commit hashes, builds locally on the server, runs migrations and health checks, and records the release. Rollback checks out the previously recorded hashes and rebuilds. There is no CI/CD pipeline, runner, private image registry, or cross-repository automation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MVP includes health checks, structured and bounded logs, safe startup, explicit migration/import separation, rollback, and encrypted daily PostgreSQL backup/restore adapted with minimal changes from Qlockify. Prometheus and Grafana are a required follow-up phase after the core project is ready and before AI work begins.
|
|
|
|
|
|
|
|
|
|
|
|
## Objective
|
|
|
|
## Objective
|
|
|
|
|
|
|
|
|
|
|
|
Provide a production-shaped, single-server hackathon platform for `zarinpal.amiirkhl.ir` that is reproducible, private by default, observable, recoverable, and reversible. The deployment repository is the operational control plane; it assembles immutable backend and frontend releases without absorbing either service's Git history or application code.
|
|
|
|
Provide a production-shaped hackathon platform that a trusted operator can understand and recover without hidden automation. A clean server can be prepared, the exact source revisions can be deployed, only approved traffic is publicly exposed, database changes and one-time data import are deliberate, failures leave useful evidence, and PostgreSQL can be restored from an encrypted off-host backup.
|
|
|
|
|
|
|
|
|
|
|
|
Success means an operator can bootstrap a clean server, deploy an exact three-repository release, prove the public and internal health contracts, run migrations and the one-time CSV import deliberately, recover PostgreSQL from an encrypted off-host backup, and roll application images back without guessing which revisions were active.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Scope
|
|
|
|
## Scope
|
|
|
|
|
|
|
|
|
|
|
|
### In scope
|
|
|
|
### MVP scope
|
|
|
|
|
|
|
|
|
|
|
|
- Production Docker Compose topology and service boundaries
|
|
|
|
- Qlockify-style nested repository layout and manual server-side deployment
|
|
|
|
- Nginx routing, TLS, canonical-host, static asset, API, and SSE behavior
|
|
|
|
- Docker Compose service topology and internal networks
|
|
|
|
- PostgreSQL, Redis, backend API, frontend, and analytics worker lifecycle
|
|
|
|
- Nginx canonical routing, security headers, TLS termination, and SSE behavior
|
|
|
|
- Liveness, readiness, startup ordering, graceful shutdown, and resource bounds
|
|
|
|
- PostgreSQL 18, Redis 7 Alpine, backend API, frontend, and worker boundaries
|
|
|
|
- Dedicated migration, one-time CSV import, and demo-credential administrative jobs
|
|
|
|
- Liveness/readiness, startup ordering, migration, shutdown, and resource limits
|
|
|
|
- Environment, configuration, secret, volume, network, and host exposure policy
|
|
|
|
- Explicit one-time CSV import and demo-credential administrative jobs
|
|
|
|
- Cross-repository CI quality gates, immutable image publication, release orchestration, and rollback
|
|
|
|
- Environment, secret, volume, and host exposure policy
|
|
|
|
- Structured logs, bounded local retention, metrics, dashboards, alerts, and release evidence
|
|
|
|
- Manual release records, health verification, and Git-hash rollback
|
|
|
|
- Encrypted PostgreSQL backup, off-host retention, guarded restore, and restore rehearsal
|
|
|
|
- Structured container logs, log rotation, Docker health status, and operator checks
|
|
|
|
- Safe first-server bootstrap and operator boundaries
|
|
|
|
- Daily encrypted PostgreSQL backup, S3-compatible upload, retention, and guarded restore adapted from Qlockify
|
|
|
|
- Ordinary frontend build consumption of the approved licensed font assets
|
|
|
|
- Safe first-server bootstrap
|
|
|
|
|
|
|
|
- Normal frontend build consumption of the tracked licensed font assets
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Required post-MVP, pre-AI phase
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Prometheus metrics collection
|
|
|
|
|
|
|
|
- Grafana dashboards
|
|
|
|
|
|
|
|
- Private exporters for host/container/PostgreSQL/Redis signals
|
|
|
|
|
|
|
|
- Symptom-based alerts and alert delivery testing
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This observability phase must be complete before the AI capability starts. It does not block the first working MVP deployment.
|
|
|
|
|
|
|
|
|
|
|
|
### Out of scope
|
|
|
|
### Out of scope
|
|
|
|
|
|
|
|
|
|
|
|
- Backend or frontend application implementation
|
|
|
|
- CI/CD pipelines, hosted/self-hosted runners, automated deploy triggers, and private image registries
|
|
|
|
- Owning ZarinPal's future production payment-write path
|
|
|
|
- Kubernetes, multi-server failover, multi-region operation, or zero-downtime claims
|
|
|
|
- Scheduled or watched CSV ingestion
|
|
|
|
- Scheduled or watched CSV ingestion
|
|
|
|
- Kubernetes, multi-region, automatic failover, or zero-downtime database migration
|
|
|
|
|
|
|
|
- Redis backup or treating Redis as a source of truth
|
|
|
|
- Redis backup or treating Redis as a source of truth
|
|
|
|
- Public administration, database, Redis, metrics, or worker endpoints
|
|
|
|
- Public database, Redis, Docker, worker, metrics, debug, or administration endpoints
|
|
|
|
- ZarinPal SSO implementation
|
|
|
|
- Backend/frontend application implementation in this repository
|
|
|
|
- Creating Compose, Nginx, Dockerfile, script, workflow, or environment artifacts before this spec is approved
|
|
|
|
- Private font injection or font copies in the deployment repository
|
|
|
|
|
|
|
|
- AI implementation
|
|
|
|
|
|
|
|
|
|
|
|
## Assumptions
|
|
|
|
## Approved assumptions and decisions
|
|
|
|
|
|
|
|
|
|
|
|
1. Production is one Linux server with sufficient CPU, memory, and SSD capacity; Compose is the accepted hackathon orchestrator.
|
|
|
|
1. The target is one supported Ubuntu server.
|
|
|
|
2. DNS for `zarinpal.amiirkhl.ir` can point directly to the server and ports 80/443 can accept ACME HTTP-01 and application traffic.
|
|
|
|
2. A dedicated trusted Ubuntu deployment user belongs to the `docker` group and runs Docker Compose. Docker-group membership is root-equivalent and is restricted to trusted operators.
|
|
|
|
3. The Git forge is Gitea-compatible and provides protected branches, Actions or an equivalent trusted runner, a private OCI registry, deployment environments, secrets, and an authenticated cross-repository event mechanism. Exact forge and runner versions remain a recorded decision.
|
|
|
|
3. The deployment repository contains ignored nested checkouts of the backend and frontend repositories, matching ADR 0001 and Qlockify.
|
|
|
|
4. Backend images contain independently runnable API, worker, migration, importer, and administrative commands from one reviewed backend revision. The importer is not the API entrypoint.
|
|
|
|
4. Normal deployment selects the latest protected `main` from all three repositories, then records the exact three commit hashes before building.
|
|
|
|
5. The frontend is a static/client-rendered SvelteKit build served by its own private web container. Nginx is the only public reverse proxy.
|
|
|
|
5. Application images are built on the production server from those recorded checkouts. They are not published to a registry.
|
|
|
|
6. PostgreSQL 18 and Redis 7 are the Qlockify-aligned initial major versions; final image tags are pinned to patch versions and immutable digests in a release manifest after compatibility tests.
|
|
|
|
6. Rollback checks out the previous recorded hashes and rebuilds. Database rollback remains constrained by migration compatibility.
|
|
|
|
7. A short maintenance window during a single-host application rollout is acceptable. The target is no more than 120 seconds of failed readiness, not a claim of zero downtime.
|
|
|
|
7. Nginx obtains and renews a Let's Encrypt certificate automatically through Certbot using HTTP-01.
|
|
|
|
8. Backup RPO is 24 hours and restore RTO is two hours for the hackathon platform.
|
|
|
|
8. PostgreSQL 18 and Redis 7 Alpine are the initial major versions. Exact patch tags are pinned before implementation.
|
|
|
|
9. The approved `IRANYekanXVFaNumVF.woff2` and required `FontLicense.txt` are frontend-owned, tracked under `static/fonts` by frontend commit `64f5ac2`, and legally distributable for this project/domain. Platform Operations neither copies nor privately injects them.
|
|
|
|
9. Redis is disposable cache/queue/coordination state and is excluded from backup. PostgreSQL remains authoritative.
|
|
|
|
|
|
|
|
10. Daily encrypted PostgreSQL backups and tested restores are MVP requirements.
|
|
|
|
|
|
|
|
11. Qlockify backup/restore/upload scripts are the implementation reference and will receive only necessary project path, service, credential, and safety changes.
|
|
|
|
|
|
|
|
12. Production `.env` files, secrets, TLS keys, Redis data, raw CSV, credentials, logs, and font assets are excluded from database backup packages.
|
|
|
|
|
|
|
|
13. Prometheus/Grafana are required after MVP readiness and before AI implementation.
|
|
|
|
|
|
|
|
14. Frontend commit `64f5ac2` owns `static/fonts/IRANYekanXVFaNumVF.woff2` and `static/fonts/FontLicense.txt`; deployment uses the normal frontend build context without another asset path.
|
|
|
|
|
|
|
|
|
|
|
|
Any invalid assumption changes this spec before implementation.
|
|
|
|
Any change to these decisions requires a spec revision before implementation.
|
|
|
|
|
|
|
|
|
|
|
|
## Architecture and trust boundaries
|
|
|
|
## Repository and server layout
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
|
|
|
/home/<deploy-user>/zarinpal-analytics-deployment/
|
|
|
|
|
|
|
|
├── .git/ deployment repository
|
|
|
|
|
|
|
|
├── backend/ ignored checkout of backend repository
|
|
|
|
|
|
|
|
│ └── .git/
|
|
|
|
|
|
|
|
├── frontend/ ignored checkout of frontend repository
|
|
|
|
|
|
|
|
│ └── .git/
|
|
|
|
|
|
|
|
├── docs/
|
|
|
|
|
|
|
|
├── backups/ ignored runtime output
|
|
|
|
|
|
|
|
└── deployment assets only after spec approval
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Each checkout keeps its own Git history and remote. Deployment operations verify the expected repository, branch, remote, clean state, and resolved absolute path before pulling, checking out, or building.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Production configuration and secrets live outside public Git. Runtime volumes, backups, certificate data, raw CSV, and generated credentials remain ignored.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Runtime architecture
|
|
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
```text
|
|
|
|
Internet
|
|
|
|
Internet
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| TCP 80/443 only
|
|
|
|
| TCP 80/443 only
|
|
|
|
v
|
|
|
|
v
|
|
|
|
+---------------- edge network ----------------+
|
|
|
|
Nginx + Certbot
|
|
|
|
| nginx (TLS, routing, limits, access logs) |
|
|
|
|
|
|
|
|
|
+--------------------+--------------------------+
|
|
|
|
+------------------------+
|
|
|
|
|
|
|
|
|
| private app network |
|
|
|
|
+-----------+-----------+
|
|
|
|
v v
|
|
|
|
| private app network |
|
|
|
|
frontend static server backend API
|
|
|
|
v v
|
|
|
|
|
|
|
|
|
frontend static web backend API
|
|
|
|
+---------+---------+
|
|
|
|
|
|
|
|
|
| private data net |
|
|
|
|
+------------+------------+
|
|
|
|
v v
|
|
|
|
| private data network |
|
|
|
|
PostgreSQL <------- Redis
|
|
|
|
v v
|
|
|
|
^ ^
|
|
|
|
PostgreSQL <--- worker ---> Redis
|
|
|
|
| |
|
|
|
|
^ ^
|
|
|
|
one-off jobs worker
|
|
|
|
| |
|
|
|
|
migrate / import / admin
|
|
|
|
one-off migrate/import/admin jobs
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Private operator path (SSH tunnel/VPN only)
|
|
|
|
|
|
|
|
-> Prometheus/Grafana and host diagnostics
|
|
|
|
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Trust boundaries are the internet-to-Nginx edge, Nginx-to-application network, application-to-data network, CI-to-private-registry boundary, deployment-runner-to-server SSH boundary, operator-to-administrative-job boundary, and backup-host-to-object-storage boundary. Assets worth protecting include merchant payment analytics, pseudonymous payer-card keys, credentials/session digests, database contents, CI/deploy keys, backup encryption keys, TLS private keys, and release provenance.
|
|
|
|
Nginx is the only public application entry point. The worker is event-driven and never scheduled. It uses Asynq pinned to `v0.26.0`, explicitly a pre-1.0 dependency. Redis loss causes controlled job/cache reconciliation and recomputation, not loss of authoritative payment or session data.
|
|
|
|
|
|
|
|
|
|
|
|
## Compose service topology
|
|
|
|
## Docker Compose topology
|
|
|
|
|
|
|
|
|
|
|
|
| Service | Image/process boundary | Networks | Persistent state | Public exposure | Restart/lifecycle |
|
|
|
|
| Service | Responsibility | Networks | State | Host exposure |
|
|
|
|
|---|---|---|---|---|---|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
| `nginx` | Pinned Nginx image with reviewed configuration | `edge`, `app` | ACME webroot and certificate volume only | `80:80`, `443:443` | `unless-stopped`; starts only after frontend/backend readiness for production traffic |
|
|
|
|
| `nginx` | TLS, canonical routing, limits, static/API/SSE proxy | `edge`, `app` | certificate mounts only | 80/443 |
|
|
|
|
| `certbot` | Pinned ACME client | `edge` | ACME account/certificate volume | None | One-shot initial issuance plus bounded renewal loop or host timer; not a health dependency after a valid certificate exists |
|
|
|
|
| `certbot` | Initial issuance and renewal | `edge` | ACME/certificate volume | none |
|
|
|
|
| `frontend` | Immutable frontend image containing static build output | `app` | None | `expose` only | `unless-stopped`; health checks a local static response |
|
|
|
|
| `frontend` | Serve static SvelteKit build | `app` | none | none |
|
|
|
|
| `backend` | Immutable backend image, API command | `app`, `data` | None | `expose` only | `unless-stopped`; waits for successful migration and healthy data dependencies |
|
|
|
|
| `backend` | Go API, sessions, authorization, analytics control | `app`, `data` | none | none |
|
|
|
|
| `worker` | Same backend digest, worker command; Asynq `v0.26.0` pinned | `data` | None | None | `unless-stopped`; event-driven only, never scheduled |
|
|
|
|
| `worker` | On-demand Asynq analytics sections | `data` | none | none |
|
|
|
|
| `postgres` | PostgreSQL 18 patch/digest | `data` | `postgres_data` | None | `unless-stopped`; graceful stop with a bounded timeout |
|
|
|
|
| `postgres` | Authoritative database | `data` | `postgres_data` | none |
|
|
|
|
| `redis` | Redis 7 patch/digest with authentication and AOF `everysec` | `data` | `redis_data`, excluded from backup | None | `unless-stopped`; routine restart persistence only, still disposable for disaster recovery |
|
|
|
|
| `redis` | Queue, cache, rate-limit, replay coordination | `data` | optional routine AOF volume, excluded from backup | none |
|
|
|
|
| `migrate` | Same backend digest, migration command | `data` | None | None | One-off; must complete successfully before backend/worker replacement |
|
|
|
|
| `migrate` | Versioned schema migration | `data` | none | one-off only |
|
|
|
|
| `import` | Same backend digest, explicit importer command/profile | `data` | Read-only operator-selected input mount | None | One-off only; never part of normal `up`, startup, cron, or health dependency |
|
|
|
|
| `import` | One-time CSV bootstrap/recovery | `data` | exact read-only input bind | one-off profile only |
|
|
|
|
| `admin` | Same backend digest, explicit credential/maintenance command/profile | `data` | Explicit output mount when requested | None | One-off only; output destination and permissions are operator-selected |
|
|
|
|
| `admin` | Demo credentials and approved maintenance | `data` | explicit output bind if requested | one-off profile only |
|
|
|
|
| `prometheus` | Pinned Prometheus image | `monitoring`, scrape access to private exporters | `prometheus_data` with seven-day/size retention | Loopback/operator path only | `unless-stopped`; platform remains functional if monitoring is down |
|
|
|
|
|
|
|
|
| `grafana` | Pinned Grafana image | `monitoring` | `grafana_data` | Loopback/operator path only | `unless-stopped`; no Nginx public route by default |
|
|
|
|
|
|
|
|
| exporters | Pinned node/container/PostgreSQL/Redis exporters as approved | `monitoring`, minimum required target network | None | None | Least-privilege, read-only metrics only |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The implementation uses the current Compose Specification, not a legacy numeric schema version. Every image uses an explicit patch tag plus digest. Floating `latest`, floating major-only application images, server-side source builds, and mutable branch references are prohibited in production.
|
|
|
|
Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate private monitoring network. Grafana binds only to loopback/private operator access unless a later ADR approves a protected public route.
|
|
|
|
|
|
|
|
|
|
|
|
### Network and exposure rules
|
|
|
|
### Image and build policy
|
|
|
|
|
|
|
|
|
|
|
|
- Only Nginx publishes host ports 80 and 443.
|
|
|
|
- PostgreSQL, Redis, Nginx, Certbot, and later monitoring images use explicit supported tags; no `latest` tag.
|
|
|
|
- PostgreSQL, Redis, backend, worker, frontend, exporters, Prometheus, and Grafana do not publish internet-facing ports.
|
|
|
|
- Backend and worker build from the same recorded backend commit and deployment Dockerfile/context contract.
|
|
|
|
- `edge`, `app`, `data`, and `monitoring` networks are separate. A service joins only the networks it needs.
|
|
|
|
- Frontend builds from the recorded frontend commit and normal build context, including its tracked font/license inputs.
|
|
|
|
- The frontend cannot reach PostgreSQL or Redis. Nginx cannot reach PostgreSQL or Redis. Exporters receive only the minimum read-only connectivity needed.
|
|
|
|
- The deployment script records resulting local image identifiers and build timestamps for diagnosis, but Git commit hashes are the reproducible source reference.
|
|
|
|
- Operator UIs bind to `127.0.0.1` when a host port is unavoidable and are reached through SSH tunneling or an approved private network.
|
|
|
|
- Build failure stops before replacing healthy running services.
|
|
|
|
- Host firewall default is deny inbound; allow established traffic, SSH from an operator allowlist, and TCP 80/443. Docker's firewall behavior must be verified from an external host, not inferred from host rules alone.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Volumes and filesystem policy
|
|
|
|
### Network and container policy
|
|
|
|
|
|
|
|
|
|
|
|
| Volume/input | Purpose | Backup | Required handling |
|
|
|
|
- Only Nginx publishes ports 80 and 443.
|
|
|
|
|---|---|---|---|
|
|
|
|
- PostgreSQL, Redis, backend, frontend, worker, and later monitoring services do not publish internet-facing ports.
|
|
|
|
| `postgres_data` | Authoritative database cluster | Logical PostgreSQL backup, not raw live-volume copying | Dedicated named volume; capacity and inode alerts |
|
|
|
|
- Frontend and Nginx cannot connect directly to PostgreSQL or Redis.
|
|
|
|
| `redis_data` | AOF for routine queue/cache restart continuity | No | Rebuildable; loss triggers idempotent run reconciliation and cold cache |
|
|
|
|
- Containers run as non-root where their official/application image supports it, drop unnecessary capabilities, set `no-new-privileges`, and use read-only filesystems where practical.
|
|
|
|
| `letsencrypt_data` | ACME account and certificate material | No database backup; separately recover/reissue | Read-write only to certbot, read-only to Nginx |
|
|
|
|
- CPU, memory, PID, database-pool, and log limits are explicit. Initial values are validated against the actual server and import/load tests.
|
|
|
|
| `certbot_webroot` | HTTP-01 challenge | No | No application files |
|
|
|
|
- Docker logs use bounded rotation so a noisy service cannot fill the server disk.
|
|
|
|
| `prometheus_data` | Short-retention metrics | No | Seven-day and size-bounded retention |
|
|
|
|
|
|
|
|
| `grafana_data` | Local dashboard state | Optional export of declarative dashboards only | Dashboards/alerts should be tracked as config after approval |
|
|
|
|
|
|
|
|
| CSV input bind | One-time archive input | Source archive retained by operator policy, outside Git | Absolute validated host path, read-only mount, checksum before import |
|
|
|
|
|
|
|
|
| admin output bind | One-time credential output if file selected | No | Absolute validated path outside repository; restrictive mode; operator removes/escrows it |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Containers use read-only root filesystems where practical, drop Linux capabilities, set `no-new-privileges`, run as non-root, and use bounded tmpfs mounts for writable runtime paths. CPU/memory/PID limits and log rotation are mandatory. Exact limits are established by load tests, with safe non-zero reservations left for PostgreSQL and the host.
|
|
|
|
## Manual deployment and release records
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Normal deployment
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Confirm exact server, deployment root, environment, available disk, recent backup, Docker health, and operator identity.
|
|
|
|
|
|
|
|
2. Acquire a deployment lock so backup/restore/import/deploy operations cannot conflict unsafely.
|
|
|
|
|
|
|
|
3. Verify all three repositories exist at their exact expected paths, use approved remotes, and contain no unexplained local changes.
|
|
|
|
|
|
|
|
4. Fetch and fast-forward each repository to its latest `origin/main`. Do not discard local/user changes automatically.
|
|
|
|
|
|
|
|
5. Record the deployment, backend, and frontend commit hashes before building.
|
|
|
|
|
|
|
|
6. Validate required configuration/secret files without printing values.
|
|
|
|
|
|
|
|
7. Render and validate Docker Compose configuration.
|
|
|
|
|
|
|
|
8. Build candidate application/Nginx images on the server.
|
|
|
|
|
|
|
|
9. Start/verify PostgreSQL and Redis.
|
|
|
|
|
|
|
|
10. Run the dedicated migration service once and require success.
|
|
|
|
|
|
|
|
11. Replace backend/worker, then frontend, then validate/reload Nginx.
|
|
|
|
|
|
|
|
12. Run public and private health/smoke checks and observe for 15 minutes.
|
|
|
|
|
|
|
|
13. Write a non-secret release record and mark it active only after checks pass.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The future deployment script adapts Qlockify's manual `deploy.sh` structure but must not use unconditional `git reset --hard`; a dirty or divergent checkout stops and asks the operator to resolve it.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Release record
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Each server-side release record contains:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- release id and UTC timestamp
|
|
|
|
|
|
|
|
- deployment, backend, and frontend Git commit hashes
|
|
|
|
|
|
|
|
- Git branch and remote identity for each checkout
|
|
|
|
|
|
|
|
- resulting local image ids/tags
|
|
|
|
|
|
|
|
- migration version
|
|
|
|
|
|
|
|
- non-secret configuration checksum
|
|
|
|
|
|
|
|
- operator identity
|
|
|
|
|
|
|
|
- preflight backup id/age
|
|
|
|
|
|
|
|
- pre/post health results
|
|
|
|
|
|
|
|
- previous release id
|
|
|
|
|
|
|
|
- outcome and rollback reason if applicable
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
It contains no secret values, `.env` content, raw CSV path/content, credentials, license text, font copy, database rows, or TLS private material.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Rollback
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Select the previous known-good release record explicitly.
|
|
|
|
|
|
|
|
- Verify the three target hashes exist, repository paths/remotes are correct, and working trees are clean.
|
|
|
|
|
|
|
|
- Check out the previous deployment/backend/frontend hashes and rebuild locally.
|
|
|
|
|
|
|
|
- Do not automatically reverse database migrations. Normal migrations use expand/contract compatibility so the previous application can run against the expanded schema.
|
|
|
|
|
|
|
|
- Run the same health/smoke checks and record rollback as a new operational event.
|
|
|
|
|
|
|
|
- Target application-only rollback time is 15 minutes under normal Git/build conditions.
|
|
|
|
|
|
|
|
|
|
|
|
## Startup, migration, import, and shutdown
|
|
|
|
## Startup, migration, import, and shutdown
|
|
|
|
|
|
|
|
|
|
|
|
### Normal release startup
|
|
|
|
### Startup order
|
|
|
|
|
|
|
|
|
|
|
|
1. Validate host identity, intended environment, clean deployment control-plane checkout, release-manifest signature/checksum, required secret files, free disk, current backup age, and Compose-rendered configuration.
|
|
|
|
1. PostgreSQL and Redis start and become healthy.
|
|
|
|
2. Pull images by digest. Do not build from server checkouts.
|
|
|
|
2. The one-off migration completes successfully.
|
|
|
|
3. Start/verify PostgreSQL and Redis health.
|
|
|
|
3. Backend and worker start and become ready.
|
|
|
|
4. Acquire a deployment lock and run the `migrate` one-off service using the candidate backend digest.
|
|
|
|
4. Frontend becomes healthy.
|
|
|
|
5. Require migration exit code zero and record migration version/duration. Application startup never runs migrations or automatic schema creation.
|
|
|
|
5. Nginx routes production traffic only when required upstreams are ready.
|
|
|
|
6. Replace backend and worker with the candidate digest. Wait for backend and worker readiness.
|
|
|
|
|
|
|
|
7. Replace frontend, then validate its local health and the font asset response from the candidate image.
|
|
|
|
|
|
|
|
8. Validate Nginx configuration, reload/recreate Nginx, and run same-origin public smoke checks.
|
|
|
|
|
|
|
|
9. Observe the release for 15 minutes. Failure thresholds trigger rollback to the previous release manifest.
|
|
|
|
|
|
|
|
10. Record the active release manifest, actor, timestamps, health evidence, and outcome.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Compose dependency conditions are a startup aid, not the deployment controller. `service_healthy` gates dependencies and `service_completed_successfully` gates migrations, but the release workflow still performs explicit pre/post checks.
|
|
|
|
Compose health/dependency conditions prevent common startup races, but the manual deployment operation still checks every step explicitly.
|
|
|
|
|
|
|
|
|
|
|
|
### Migration contract
|
|
|
|
### Migration contract
|
|
|
|
|
|
|
|
|
|
|
|
- Migrations are reviewed, versioned backend artifacts run exactly once per release under an advisory/exclusive migration lock.
|
|
|
|
- Application startup never performs migrations or automatic schema creation.
|
|
|
|
- A migration must be backward-compatible with the currently running application for the rollback window: expand first, deploy readers/writers, clean up only in a later approved release.
|
|
|
|
- Migrations run from the recorded candidate backend revision under one migration lock.
|
|
|
|
- Destructive, long-locking, or non-reversible migrations require an explicit operator approval, fresh verified backup, measured lock/duration evidence on production-shaped data, and a dedicated rollback/recovery plan.
|
|
|
|
- Changes are backward-compatible through expand/deploy/contract sequencing.
|
|
|
|
- Rollback never automatically runs a down migration. The default rollback restores prior application images against the expanded compatible schema.
|
|
|
|
- Destructive or long-locking migrations require explicit approval, a fresh verified backup, measured evidence on production-shaped data, and a separate recovery plan.
|
|
|
|
- A failed migration stops the release before application replacement. It must not be retried blindly when partial effects are possible.
|
|
|
|
- Migration failure stops deployment before candidate backend/worker replacement.
|
|
|
|
|
|
|
|
- Database down-migration is never automatic.
|
|
|
|
|
|
|
|
|
|
|
|
### One-time CSV import contract
|
|
|
|
### One-time CSV import
|
|
|
|
|
|
|
|
|
|
|
|
- Import is an explicit operator action after schema migration, never a Compose startup command, scheduled service, file watcher, or deployment trigger.
|
|
|
|
- Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
|
|
|
|
- The operator supplies an absolute input path outside Git. The deployment wrapper resolves and validates that path, mounts exactly the file read-only, and records its SHA-256 through the backend import manifest.
|
|
|
|
- The operator provides an exact resolved path outside Git; only that file is mounted read-only.
|
|
|
|
- Import runs with one concurrency slot and an import lock. It has bounded database resources and cannot overlap schema migration, restore, or another import.
|
|
|
|
- Import cannot overlap migration, restore, or another import.
|
|
|
|
- Re-running an identical completed checksum/schema version is a safe no-op or explicit verified reconciliation according to the approved Data Foundation spec.
|
|
|
|
- The backend importer validates checksum/schema, streams data, reconciles counts, and records incomplete failure safely according to the Data Foundation spec.
|
|
|
|
- Failure leaves an incomplete import run with evidence; it never marks data complete. Resume/retry behavior is owned by the backend import contract.
|
|
|
|
- An identical completed import is idempotent and does not duplicate business rows.
|
|
|
|
- Demo credential generation is a separate `admin` job after a successful import. Plaintext output never enters Compose logs, CI logs, Git, or the release manifest.
|
|
|
|
- Demo credential generation is a separate one-time `admin` job whose plaintext output never enters Compose logs, Git, backups, or shell history by default.
|
|
|
|
|
|
|
|
|
|
|
|
### Shutdown
|
|
|
|
### Graceful shutdown
|
|
|
|
|
|
|
|
|
|
|
|
- Nginx stops accepting new traffic before backend termination; backend and worker receive a termination signal and a bounded drain period.
|
|
|
|
- Nginx stops accepting new traffic before backend termination.
|
|
|
|
- Worker shutdown stops fetching new Asynq jobs and allows an active job to finish only within its existing 60-second attempt timeout. Unfinished retryable work remains/reappears through queue semantics.
|
|
|
|
- Backend drains within a bounded timeout.
|
|
|
|
- PostgreSQL stops last. Forced kill is a failure condition recorded by the deploy workflow.
|
|
|
|
- Worker stops fetching new jobs; active attempts remain bounded by their existing 60-second timeout and retry rules.
|
|
|
|
|
|
|
|
- PostgreSQL stops last. Forced termination is recorded as a deployment failure.
|
|
|
|
|
|
|
|
|
|
|
|
## Health and readiness contracts
|
|
|
|
## Health and readiness
|
|
|
|
|
|
|
|
|
|
|
|
Health responses are generic and contain no versions, dependency addresses, credentials, stack traces, tenant information, or database details.
|
|
|
|
Health responses are generic and never reveal versions, addresses, schema details, credentials, tenant data, or stack traces.
|
|
|
|
|
|
|
|
|
|
|
|
| Component | Liveness | Readiness | Failure meaning |
|
|
|
|
| Component | Liveness | Readiness |
|
|
|
|
|---|---|---|---|
|
|
|
|
|---|---|---|
|
|
|
|
| Nginx | Local config/process endpoint answers | TLS virtual host can serve frontend and proxy backend readiness | Restart for failed liveness; remove public readiness on upstream failure |
|
|
|
|
| Nginx | process/config responds | TLS virtual host can serve frontend and proxy backend |
|
|
|
|
| Frontend | Local static sentinel returns 200 | `index.html`, hashed assets, and required font file are readable | Candidate image/build is incomplete |
|
|
|
|
| Frontend | local static sentinel | index and required built assets/font are readable |
|
|
|
|
| Backend | Process/event loop answers without external dependency calls | PostgreSQL reachable, expected migration present, Redis reachable for analytics/session rate limits, and startup validation complete | Do not route new API traffic; do not restart solely for dependency outage |
|
|
|
|
| Backend | process responds without dependency calls | expected migration, PostgreSQL, and Redis capabilities are available |
|
|
|
|
| Worker | Worker process answers local probe | PostgreSQL/Redis reachable, queue consumer registered, configured concurrency valid, heartbeat fresh | Stop accepting/claiming jobs until ready |
|
|
|
|
| Worker | process probe | PostgreSQL/Redis available, queue consumer registered, heartbeat current |
|
|
|
|
| PostgreSQL | Container process | `pg_isready` against the configured database/user plus application-level migration check from backend | Dependency unavailable or schema mismatch |
|
|
|
|
| PostgreSQL | process | authenticated `pg_isready` for configured database/user |
|
|
|
|
| Redis | Process responds to authenticated `PING` | Required queue/cache commands usable and memory policy validated | Analytics/cache coordination unavailable |
|
|
|
|
| Redis | process | authenticated `PING` and required queue/cache commands |
|
|
|
|
|
|
|
|
|
|
|
|
Defaults after implementation:
|
|
|
|
Initial health timing target is 10-second interval, 3-second timeout, 30-second start period, and five failures before unhealthy, revised only from measured startup evidence. Dependency failure changes readiness but does not deliberately create liveness restart storms.
|
|
|
|
|
|
|
|
|
|
|
|
- Health interval: 10 seconds; timeout: 3 seconds; start period: 30 seconds; five consecutive failures before unhealthy, adjusted only from observed cold-start data.
|
|
|
|
## Nginx, domain, SSE, and TLS
|
|
|
|
- Public `/health/live` and `/health/ready` return only generic status. Deep dependency details remain in private metrics/logs.
|
|
|
|
|
|
|
|
- Liveness never depends on PostgreSQL, Redis, or an external service; dependency outages must not create restart storms.
|
|
|
|
|
|
|
|
- Readiness fails closed for a schema mismatch and for dependencies required to serve the routed capability.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Nginx, domain, and TLS contract
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Routing
|
|
|
|
### Routing
|
|
|
|
|
|
|
|
|
|
|
|
- Port 80 serves `/.well-known/acme-challenge/` and redirects all other requests to `https://zarinpal.amiirkhl.ir$request_uri` with a permanent redirect.
|
|
|
|
- Port 80 serves `/.well-known/acme-challenge/` and redirects everything else to `https://zarinpal.amiirkhl.ir$request_uri`.
|
|
|
|
- Any alternate hostname redirects to the canonical hostname. Unknown hosts are rejected by a default server and never proxied.
|
|
|
|
- Alternate approved hostnames redirect to the canonical hostname. Unknown hosts are rejected by a default server.
|
|
|
|
- `/api/` proxies to the backend over the private `app` network with the original host, verified client address chain, scheme, and a generated/preserved request ID.
|
|
|
|
- `/api/` proxies only to the backend with host, scheme, safe client-address chain, and request id.
|
|
|
|
- The exact authenticated analytics SSE path is generated from the approved API contract and receives a dedicated location block; it is never matched only by a broad accidental prefix.
|
|
|
|
- `/` serves/proxies the frontend; SPA fallback never captures API errors.
|
|
|
|
- `/` proxies to the private static frontend container. The frontend container owns SPA fallback behavior so `/api/` can never fall through to `index.html`.
|
|
|
|
- Fingerprinted assets receive long immutable caching; HTML/mutable manifests revalidate.
|
|
|
|
- Fingerprinted static assets use a one-year immutable cache. HTML and mutable manifests use `no-cache`/revalidation. Font MIME type, `Content-Type`, and same-origin caching are verified.
|
|
|
|
- Public metrics, debug, profiling, database, Redis, worker, Docker, or administration routes do not exist.
|
|
|
|
- Public API documentation, metrics, debug, profiling, database administration, Redis, worker, and Grafana routes are absent unless a later ADR explicitly exposes an authenticated operator path.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### SSE route
|
|
|
|
### SSE
|
|
|
|
|
|
|
|
|
|
|
|
- Use HTTP/1.1 upstream, clear hop-by-hop `Connection`, preserve `Cookie`, `Last-Event-ID`, request ID, host, scheme, and client-forwarding headers.
|
|
|
|
- The approved exact analytics SSE route gets a dedicated location.
|
|
|
|
- Disable proxy buffering, proxy cache, response compression, and transformations for this route; send `X-Accel-Buffering: no` as defense in depth.
|
|
|
|
- Upstream HTTP/1.1 is used; `Cookie`, `Last-Event-ID`, request id, host, scheme, and client-forwarding headers are preserved.
|
|
|
|
- Set read timeout safely above the 15-second heartbeat (initial target 75 seconds) and a bounded connect/send timeout. Do not inherit a one-hour general API timeout.
|
|
|
|
- Proxy buffering, caching, compression, and transformations are disabled for the route.
|
|
|
|
- Do not cache authenticated SSE responses. Reauthorize every connect/replay request; Nginx does not bypass backend session checks.
|
|
|
|
- Read timeout is safely above the 15-second heartbeat; initial target is 75 seconds.
|
|
|
|
- Enforce connection/body/header limits without buffering the event stream. Backend user/merchant SSE concurrency limits remain authoritative.
|
|
|
|
- Every connection/replay is authenticated by the backend. Nginx does not bypass session or tenant checks.
|
|
|
|
|
|
|
|
|
|
|
|
### TLS and HTTP security
|
|
|
|
### TLS
|
|
|
|
|
|
|
|
|
|
|
|
- TLS 1.2 and 1.3 only, modern cipher defaults, session reuse, and automated certificate renewal.
|
|
|
|
- Certbot obtains the certificate through HTTP-01 after DNS and port checks.
|
|
|
|
- ACME production issuance occurs only after DNS and staging issuance checks pass. Nginx may use a temporary self-signed certificate only during isolated bootstrap; it must not be reported as production-ready.
|
|
|
|
- Staging issuance/renewal is tested before production issuance.
|
|
|
|
- Certificate expiry warning at 21 days and page at seven days. Renewal is testable without waiting for expiry.
|
|
|
|
- Nginx may temporarily use a self-signed certificate only during isolated first bootstrap; it is never production-ready.
|
|
|
|
- Enable HSTS only after a valid renewable certificate is proven; initial target `max-age=31536000; includeSubDomains` only if subdomain ownership is confirmed. Preload requires separate approval.
|
|
|
|
- TLS 1.2/1.3 only, with reviewed security headers.
|
|
|
|
- Apply `X-Content-Type-Options: nosniff`, a restrictive frame policy, referrer policy, permissions policy, and the frontend-approved CSP. Security headers are tested on HTML, API, error, and redirect responses.
|
|
|
|
- HSTS is enabled only after production issuance and renewal are proven. Preload requires separate approval.
|
|
|
|
- Login and API request-body limits are small and explicit. The CSV archive is never uploaded through public Nginx.
|
|
|
|
- Certificate validity/expiry is checked daily in MVP operations; Prometheus alerting is added in the required post-MVP observability phase.
|
|
|
|
|
|
|
|
|
|
|
|
## Configuration and secret handling
|
|
|
|
## Configuration and secrets
|
|
|
|
|
|
|
|
|
|
|
|
### Configuration classes
|
|
|
|
- A committed safe example names required variables and contains placeholders only.
|
|
|
|
|
|
|
|
- Real production `.env` and secret files live outside public Git with mode 0600 or stricter.
|
|
|
|
|
|
|
|
- Secrets are mounted only into the services that need them and never passed in command-line arguments or embedded in images.
|
|
|
|
|
|
|
|
- Frontend receives no production secret.
|
|
|
|
|
|
|
|
- Compose validation and diagnostics must not print resolved secret values.
|
|
|
|
|
|
|
|
- Backup object-store credentials and encryption passphrase are read from protected server configuration, following the Qlockify script interface with renamed ZarinPal variables/paths as needed.
|
|
|
|
|
|
|
|
- If a secret reaches Git/logs, rotate/revoke it first, then clean history/evidence as a separate incident action.
|
|
|
|
|
|
|
|
|
|
|
|
| Class | Examples | Storage/delivery |
|
|
|
|
### Licensed font
|
|
|
|
|---|---|---|
|
|
|
|
|
|
|
|
| Public tracked defaults | Domain placeholder, port names, timeouts, concurrency defaults, metric retention | `.env.example`/documented Compose defaults after approval; no real endpoints or credentials |
|
|
|
|
|
|
|
|
| Environment-specific non-secret config | Production domain, image digests, resource limits, alert destinations by alias | Signed/private release manifest or protected forge variables |
|
|
|
|
|
|
|
|
| Runtime secrets | PostgreSQL/Redis passwords, session/CSRF keys, backup credentials/key, deploy key, Grafana admin credential | Forge/host secret store; mounted as read-only files where supported |
|
|
|
|
|
|
|
|
| TLS private material | ACME account and private keys | Restricted certificate volume; never environment variables or Git |
|
|
|
|
|
|
|
|
| Licensed frontend font inputs | `static/fonts/IRANYekanXVFaNumVF.woff2`, `static/fonts/FontLicense.txt` | Tracked and consumed in the frontend repository's ordinary build context; not copied into deployment Git |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Requirements:
|
|
|
|
Frontend commit `64f5ac2` tracks `static/fonts/IRANYekanXVFaNumVF.woff2` and `static/fonts/FontLicense.txt`. The normal frontend server build includes them. Deployment adds no fetch, injection, credential, duplicate binary, or license copy. Smoke checks verify the emitted font resource and content type without reproducing license text.
|
|
|
|
|
|
|
|
|
|
|
|
- Production secret files live outside the checkout under a root/operator-controlled directory such as `/etc/zarinpal-analytics/secrets`, mode 0600 or stricter, and are mounted only into consumers.
|
|
|
|
## MVP logging and operator checks
|
|
|
|
- Compose interpolation files contain non-secret release/config values where possible. Secrets are not passed on command lines, embedded in image layers, printed by `compose config`, or included in diagnostic bundles.
|
|
|
|
|
|
|
|
- A committed example names every required variable and supplies safe placeholders/validation rules, never plausible credentials.
|
|
|
|
|
|
|
|
- Backend and worker share only the secrets each command needs. Frontend receives no runtime secret.
|
|
|
|
|
|
|
|
- CI test credentials are isolated from production. The deploy job gets no database or backup decryption key.
|
|
|
|
|
|
|
|
- Secret rotation is documented and can be performed without rebuilding unrelated images. If a secret enters Git or logs, revoke/rotate first, then purge as a separate incident task.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Font build dependency
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The approved decision, implemented in frontend commit `64f5ac2`, places `IRANYekanXVFaNumVF.woff2` and `FontLicense.txt` in the frontend repository under `static/fonts`. Platform Operations creates no separate injection workflow or deployment-repository copy. Frontend CI verifies both required tracked files exist, the production build emits the expected font resource, and the candidate container serves it with the correct MIME type. Deployment records only the frontend Git SHA and immutable image digest; the font travels as part of that ordinary frontend artifact.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Cross-repository CI/CD and release provenance
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Service repository triggers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Backend pull request:** format/lint, unit/integration tests, race tests, `go vet`, `golangci-lint`, `govulncheck`, migration verification, importer fixtures, queue/retry/recovery tests against pinned pre-1.0 Asynq `v0.26.0`, image build, image scan, and SBOM/provenance generation. No production deploy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Backend protected-main/tag:** after all gates pass, build once, publish the immutable backend image to the private registry, capture its digest plus backend SHA/migration compatibility metadata, and send an authenticated candidate event to the deployment repository.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Frontend pull request:** frozen dependency install, format/lint, `svelte-check`, unit/component tests, Playwright critical flows, accessibility/RTL/reduced-motion checks, production build, required font/license file presence, image build, scan, and SBOM/provenance generation. No production deploy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Frontend protected-main/tag:** after all gates pass, build once, publish an immutable frontend image, capture digest plus frontend SHA/API compatibility metadata, and send an authenticated candidate event to the deployment repository.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Deployment pull request:** documentation checks plus, after implementation is approved, Compose render/validation, Nginx syntax tests, shell lint/tests, secret scanning, policy checks for published ports/digests/health checks, backup/restore tests, and integration smoke using explicitly selected backend/frontend candidate digests.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Deployment protected-main/tag/manual release:** assemble or approve a release manifest, acquire a single production deployment lock, run preflight/migration/deploy/postflight, and monitor. A service event creates a candidate; it does not independently SSH to production or mutate the server.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Release manifest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Every production release records:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- release id and creation timestamp
|
|
|
|
|
|
|
|
- deployment, backend, and frontend Git SHAs
|
|
|
|
|
|
|
|
- immutable image names and digests for every service
|
|
|
|
|
|
|
|
- PostgreSQL/Redis/Nginx/monitoring image digests
|
|
|
|
|
|
|
|
- expected migration version and compatibility direction
|
|
|
|
|
|
|
|
- Compose/config schema revision and non-secret config checksum
|
|
|
|
|
|
|
|
- SBOM/provenance references and vulnerability-gate result
|
|
|
|
|
|
|
|
- actor/approval, preflight backup id, deploy outcome, and previous release id
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The manifest contains no secret values, license identifier, raw font copy, credentials, CSV path/content, or database data. It is stored in the deployment release history and on the server as an append-only operational record. Production deploys consume digests from this manifest and never `git reset --hard` nested service checkouts or rebuild source on the server.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Release gate and rollback
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Only protected immutable revisions can become candidates. Required checks and one human production approval are mandatory.
|
|
|
|
|
|
|
|
- Deployments are serialized; overlapping backend/frontend/deployment events coalesce into a newly reviewed manifest rather than racing.
|
|
|
|
|
|
|
|
- Pre-deploy requires a successful backup within 24 hours; a destructive migration requires a fresh on-demand verified backup.
|
|
|
|
|
|
|
|
- Post-deploy gates: Compose health, TLS/canonical redirect, static/font response, login page, generic liveness/readiness, authenticated smoke with a dedicated non-sensitive account, one analytics start/SSE heartbeat, worker/queue health, and no threshold breach during the 15-minute watch.
|
|
|
|
|
|
|
|
- Application/config failure rolls back to the previous manifest's image digests and compatible config. The failed candidate remains recorded.
|
|
|
|
|
|
|
|
- Database down-migration is never automatic. If a migration is not backward-compatible, release approval requires a separate database recovery plan and extended outage acknowledgment.
|
|
|
|
|
|
|
|
- Rollback completes within 15 minutes for an application-only release under normal registry/network conditions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Observability and reliability objectives
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Operator questions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Telemetry must answer:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Can users reach, authenticate, and load the dashboard through the canonical TLS endpoint?
|
|
|
|
|
|
|
|
2. Which release and request/run/section caused a failure, and was PostgreSQL, Redis, queueing, calculation, or SSE delivery responsible?
|
|
|
|
|
|
|
|
3. Are queue delay, calculation duration, database saturation, Redis memory, disk, certificate age, or backup age approaching a user-visible limit?
|
|
|
|
|
|
|
|
4. Can the current backup be restored to a clean destination within the stated RTO?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Logs
|
|
|
|
### Logs
|
|
|
|
|
|
|
|
|
|
|
|
- Application and worker logs are structured JSON to stdout with stable event names, UTC timestamp, level, service, release id, request id, analytics run/section/job id where applicable, safe failure class, and duration.
|
|
|
|
- Backend and worker emit structured JSON to stdout with UTC timestamp, level, service, release id, request id, analytics run/section/job ids where applicable, safe error class, and duration.
|
|
|
|
- Nginx access logs are structured and include request id, route template/category, status, bytes, request/upstream duration, and upstream status. They exclude cookies, authorization headers, query strings that may contain sensitive values, and response bodies.
|
|
|
|
- Nginx access logs include request id, safe route, status, bytes, request/upstream duration, and upstream status.
|
|
|
|
- Never log credentials, raw session tokens/cookies, password hashes, payer-card keys, raw CSV rows, database URLs, Redis URLs/passwords, backup keys, or full merchant payloads.
|
|
|
|
- Never log passwords, hashes, cookies, raw tokens, payer-card keys, raw CSV rows, database/Redis URLs, backup keys, `.env` contents, or full merchant payloads.
|
|
|
|
- Docker log storage uses a bounded driver/rotation policy. Initial target is 10 MiB per file, three files per container; validate against incident needs and disk budget.
|
|
|
|
- Docker log rotation is explicit; initial target is 10 MiB per file and three files per container, adjusted from incident/disk evidence.
|
|
|
|
- Production debug logging is off by default and time-bounded when enabled. Logs retain seven local days or less under a size cap; longer retention requires an approved privacy/cost decision.
|
|
|
|
- Production debug logging is off by default and time-bounded when enabled.
|
|
|
|
|
|
|
|
|
|
|
|
### Metrics and dashboards
|
|
|
|
### MVP manual checks
|
|
|
|
|
|
|
|
|
|
|
|
- HTTP RED metrics by bounded route template/method/status class; latency histograms expose p50/p95/p99.
|
|
|
|
The operator can check:
|
|
|
|
- Backend dependency RED metrics for PostgreSQL and Redis without query text, merchant id, request id, raw URL, or error text as labels.
|
|
|
|
|
|
|
|
- Worker: queue depth/oldest age, active jobs, global/per-merchant saturation as bounded aggregate series, section duration, retry/failure class, cache hit/miss, SSE connections/reconnect/replay/snapshot fallback, and suppression rate.
|
|
|
|
|
|
|
|
- PostgreSQL: availability, connections/pool saturation, transaction/lock pressure, slow-query aggregate, database/volume size, replication metric only if replication exists.
|
|
|
|
|
|
|
|
- Redis: availability, memory/limit, eviction, rejected connection, persistence status, queue depth, and key expiry aggregate.
|
|
|
|
|
|
|
|
- Host/container: CPU, memory, load, filesystem space/inodes, OOM/restarts, network, and certificate days remaining.
|
|
|
|
|
|
|
|
- Backup: last success timestamp, duration, size, remote upload/checksum status, retention count, and last successful restore rehearsal.
|
|
|
|
|
|
|
|
- Metric labels are bounded; merchant/user/card/request identifiers never become metric labels.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Prometheus and Grafana remain private. Dashboards show service overview, release health, analytics worker/SSE, PostgreSQL/Redis capacity, host capacity, and backup/TLS status. Alerts link to a runbook and use only `page` or `ticket` severity.
|
|
|
|
- Compose health and restart counts
|
|
|
|
|
|
|
|
- Nginx/backend/worker/PostgreSQL/Redis logs
|
|
|
|
|
|
|
|
- public TLS/canonical redirect and generic health
|
|
|
|
|
|
|
|
- disk space/inodes and database/backup size
|
|
|
|
|
|
|
|
- Redis memory and queue health
|
|
|
|
|
|
|
|
- certificate days remaining and renewal result
|
|
|
|
|
|
|
|
- last verified backup age and retention count
|
|
|
|
|
|
|
|
- current/previous release records
|
|
|
|
|
|
|
|
|
|
|
|
### Initial SLOs and alert thresholds
|
|
|
|
An operator checklist/runbook provides exact commands after implementation approval.
|
|
|
|
|
|
|
|
|
|
|
|
| Objective | Target / measurement | Initial alert |
|
|
|
|
## Required post-MVP observability phase
|
|
|
|
|---|---|---|
|
|
|
|
|
|
|
|
| Public availability | 99.5% successful synthetic HTTPS readiness over rolling 7 days, excluding approved maintenance | Page after 5 minutes of consecutive failure |
|
|
|
|
|
|
|
|
| Non-streaming API errors | <1% unexpected 5xx over 5 minutes | Page at >2% for 5 minutes with minimum traffic |
|
|
|
|
|
|
|
|
| Non-analytics API latency | p95 <500 ms over 5 minutes under acceptance load | Ticket at p95 >500 ms for 15 minutes; page at >2 s for 5 minutes |
|
|
|
|
|
|
|
|
| Analytics start response | p95 <1 second, work queued/deduplicated before response | Page at p95 >3 seconds or >2% 5xx for 5 minutes |
|
|
|
|
|
|
|
|
| SSE delivery | Connection established <2 seconds p95; heartbeat gap <=20 seconds while healthy | Page when synthetic stream misses two heartbeats |
|
|
|
|
|
|
|
|
| Section jobs | Queue delay p95 <5 seconds; attempts always <=60 seconds; no more than two transient retries | Ticket if oldest queue age >30 seconds; page >2 minutes |
|
|
|
|
|
|
|
|
| Capacity | Host/data volumes keep >=20% and >=10 GiB free; no sustained DB/Redis saturation | Ticket below 25%; page below 15% or projected exhaustion <24 hours |
|
|
|
|
|
|
|
|
| Backup | Encrypted off-host backup succeeds every 24 hours | Page when last verified remote backup age >30 hours |
|
|
|
|
|
|
|
|
| Restore | Monthly clean-destination rehearsal completes within two hours | Ticket immediately on rehearsal failure |
|
|
|
|
|
|
|
|
| TLS | Valid certificate and automated renewal | Ticket <21 days; page <7 days |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Targets are hackathon baselines, not evidence of current performance. Load and failure tests must validate or revise them before approval for production use.
|
|
|
|
Before AI implementation begins, add private Prometheus/Grafana and approved exporters/instrumentation for:
|
|
|
|
|
|
|
|
|
|
|
|
## Backup, restore, and disaster recovery
|
|
|
|
- HTTP RED metrics with bounded route/status labels and latency histograms
|
|
|
|
|
|
|
|
- worker queue depth/oldest age, section duration, retries/failures, concurrency, cache hit/miss, and SSE connection/replay behavior
|
|
|
|
|
|
|
|
- PostgreSQL availability, connections, locks, query duration aggregates, database/volume size
|
|
|
|
|
|
|
|
- Redis availability, memory, eviction, persistence, connections, and queue state
|
|
|
|
|
|
|
|
- host/container CPU, memory, filesystem/inodes, OOM/restarts, and network
|
|
|
|
|
|
|
|
- backup success/age/duration/size/remote verification and restore-rehearsal age
|
|
|
|
|
|
|
|
- TLS expiry and renewal status
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Grafana and Prometheus remain private. Alerts are symptom-based, actionable, link to runbooks, and use only page/ticket severity. Merchant/user/request/card ids never become metric labels.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Initial targets to validate:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Objective | Target |
|
|
|
|
|
|
|
|
|---|---|
|
|
|
|
|
|
|
|
| Public availability | 99.5% over rolling 7 days excluding approved maintenance |
|
|
|
|
|
|
|
|
| Unexpected API 5xx | under 1% over 5 minutes under acceptance traffic |
|
|
|
|
|
|
|
|
| Non-analytics API latency | p95 under 500 ms under acceptance load |
|
|
|
|
|
|
|
|
| Analytics start | p95 under 1 second |
|
|
|
|
|
|
|
|
| SSE | connection p95 under 2 seconds; healthy heartbeat gap at most 20 seconds |
|
|
|
|
|
|
|
|
| Section jobs | queue delay p95 under 5 seconds; every attempt at most 60 seconds |
|
|
|
|
|
|
|
|
| Capacity | at least 20% and 10 GiB free on critical filesystems |
|
|
|
|
|
|
|
|
| Backup | verified encrypted off-host backup every 24 hours |
|
|
|
|
|
|
|
|
| Restore | clean rehearsal within two hours |
|
|
|
|
|
|
|
|
| TLS | valid automatically renewable certificate |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Backup and restore
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Qlockify adaptation boundary
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Reuse and minimally adapt these read-only reference scripts after implementation approval:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `qlockify-deployment/scripts/backup.sh`
|
|
|
|
|
|
|
|
- `qlockify-deployment/scripts/backup-upload-s3.sh`
|
|
|
|
|
|
|
|
- `qlockify-deployment/scripts/restore.sh`
|
|
|
|
|
|
|
|
- `qlockify-deployment/scripts/restore-from-s3.sh`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Necessary changes include deployment root, archive/prefix names, Compose service names, protected credentials, PostgreSQL-only contents, manifest/checksums, exact target validation, and production confirmation guards. The core `pg_dump`/archive, OpenSSL encryption, rclone S3-compatible upload, local/remote retention, download/decrypt, and restore flow remains recognizable.
|
|
|
|
|
|
|
|
|
|
|
|
### Backup contract
|
|
|
|
### Backup contract
|
|
|
|
|
|
|
|
|
|
|
|
- PostgreSQL is the only required application data backup. Redis, Prometheus, Grafana runtime state, TLS material, environment files, secrets, source CSV, and licensed font files are excluded.
|
|
|
|
- Daily PostgreSQL logical dump using a matching PostgreSQL 18 client.
|
|
|
|
- Run a daily PostgreSQL logical backup using the PostgreSQL 18 client matching the server major. Use custom/directory format suitable for `pg_restore`, include globals/roles in a separately controlled form if required, and fail on command errors.
|
|
|
|
- Backup package contains database archive plus non-secret manifest/checksums only.
|
|
|
|
- A backup package contains the database archive, checksums, PostgreSQL/server version, schema migration version, release id, start/end timestamps, size/count evidence, and a format version. It contains no environment files or secret values.
|
|
|
|
- Encrypt before durable local/remote storage using the approved Qlockify-compatible mechanism.
|
|
|
|
- Encrypt before durable local or remote storage with an authenticated, approved scheme. The decryption key is stored separately from the server and object-store credentials.
|
|
|
|
- Upload to private S3-compatible storage over TLS and verify the remote object.
|
|
|
|
- Upload to private S3-compatible object storage over TLS. Enable provider-side versioning/object lock when available.
|
|
|
|
- Retain the latest three verified encrypted local backups and seven verified encrypted remote backups.
|
|
|
|
- Initial retention aligns with Qlockify operational practice: latest three encrypted local backups and latest seven encrypted remote backups. Retention deletion runs only after successful upload and checksum verification.
|
|
|
|
- Delete stale backups only after a new remote backup is verified; never delete the last known-good copy.
|
|
|
|
- A successful dump alone is not a successful backup: archive listing/checksum, remote object verification, and manifest completion are required.
|
|
|
|
- Exclude `.env`, secrets, Redis, TLS keys, raw CSV, credentials, logs, monitoring state, font binary, and `FontLicense.txt`.
|
|
|
|
|
|
|
|
- A dump is successful only after archive listing/checksum, encryption, upload, remote verification, and manifest completion.
|
|
|
|
|
|
|
|
|
|
|
|
### Restore contract
|
|
|
|
### Restore contract
|
|
|
|
|
|
|
|
|
|
|
|
- Restore defaults to a new empty database on a non-production destination. PostgreSQL documents that dumps can execute source-controlled SQL; only trusted project backups may be restored.
|
|
|
|
- Default restore target is a new empty non-production database.
|
|
|
|
- Before extraction/decryption, validate file type, checksum, manifest schema, expected object key, available space, PostgreSQL compatibility, and exact resolved destination.
|
|
|
|
- Validate explicit backup id, checksums, manifest, archive listing, PostgreSQL compatibility, free space, and exact resolved destination before mutation.
|
|
|
|
- Production restore requires all of: explicit environment flag, exact canonical hostname, exact database name, typed one-time confirmation phrase containing the backup id, maintenance mode, active deployment lock, current-state backup when feasible, and an operator audit record.
|
|
|
|
- Production restore additionally requires environment, host, database, backup id, typed confirmation, maintenance mode, exclusive lock, and named operator.
|
|
|
|
- Never restore over a running production database in place. Stop writers, terminate sessions deliberately, restore into a newly created empty database/cluster, validate, then switch connectivity according to the approved implementation runbook.
|
|
|
|
- Never pipe an unchecked dump into the active database or restore over a running destination in place.
|
|
|
|
- Use `pg_restore` with error-stop semantics and appropriate ownership handling. Run migrations only if the selected release contract requires them; never silently migrate beyond the chosen restore point.
|
|
|
|
- Restore into an empty database/cluster, validate counts/invariants/tenant isolation/login/cold Redis analytics/SSE, then switch/reopen traffic.
|
|
|
|
- Post-restore validation checks database/schema/import manifests, representative counts, tenant isolation smoke, login/session behavior, analytics cold-cache recomputation, and Redis-empty recovery before reopening traffic.
|
|
|
|
- Rehearse monthly and before destructive migrations; record achieved RPO/RTO.
|
|
|
|
- Rehearse restore monthly and before a destructive migration. Record RPO/RTO achieved, backup id, release id, validation results, and remediation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Disaster scenarios
|
|
|
|
## Safe first-server bootstrap
|
|
|
|
|
|
|
|
|
|
|
|
| Failure | Expected response |
|
|
|
|
1. Record provider/instance, OS, network, capacity, recovery console, operators, DNS, and rollback window.
|
|
|
|
|
|
|
|
2. Prove two key-only operator logins and provider console recovery before disabling root/password SSH.
|
|
|
|
|
|
|
|
3. Configure default-deny firewall: operator-allowlisted SSH plus public 80/443 only.
|
|
|
|
|
|
|
|
4. Patch the OS, enable time sync/security updates/log rotation/disk monitoring, and reboot-test access.
|
|
|
|
|
|
|
|
5. Install Docker Engine/Compose from the approved source and add only the dedicated trusted deploy user to the `docker` group.
|
|
|
|
|
|
|
|
6. Clone the deployment repository and backend/frontend nested repositories into exact validated paths.
|
|
|
|
|
|
|
|
7. Create protected configuration/secret paths outside Git and validate permissions.
|
|
|
|
|
|
|
|
8. Validate Compose and local builds without starting public traffic.
|
|
|
|
|
|
|
|
9. Point DNS, test ACME staging, issue production TLS, and verify canonical routing.
|
|
|
|
|
|
|
|
10. Start data services, run migration, then API/worker/frontend/Nginx and health checks.
|
|
|
|
|
|
|
|
11. Run the explicit one-time CSV import and separate credential generation only after the platform is healthy.
|
|
|
|
|
|
|
|
12. Complete encrypted remote backup and clean restore rehearsal before handoff.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Threats and failure behavior
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Failure/threat | Required behavior |
|
|
|
|
|---|---|
|
|
|
|
|---|---|
|
|
|
|
| Redis loss/corruption | Stop worker if unsafe, replace empty Redis, reconcile stale analytics runs idempotently, recompute cache; no PostgreSQL restore |
|
|
|
|
| Wrong server/path/repository | Stop before Git, build, volume, or database mutation |
|
|
|
|
| Frontend/backend image failure | Roll back to previous release manifest digests; no database restore |
|
|
|
|
| Dirty/diverged checkout | Stop and ask operator; never unconditional hard reset |
|
|
|
|
| Bad backward-compatible migration | Roll back app image, retain expanded schema, ship corrective migration |
|
|
|
|
| Pull/build failure | Keep current containers/release; record evidence |
|
|
|
|
| PostgreSQL logical corruption/operator error | Enter maintenance, select trusted backup, guarded restore to empty destination, validate, then reopen |
|
|
|
|
| Missing/malformed secret/config | Fail before mutation without printing value |
|
|
|
|
| Host loss | Bootstrap replacement host, restore secrets through separate escrow, deploy chosen manifest, restore PostgreSQL, issue/restore TLS, run validation |
|
|
|
|
| PostgreSQL/Redis unhealthy | Do not start/route dependent candidate services |
|
|
|
|
| Object-store outage | Preserve encrypted local retention, alert, retry bounded upload; do not delete the last verified local backup |
|
|
|
|
| Migration failure | Stop release; no candidate backend/worker; no automatic down migration |
|
|
|
|
| Certificate renewal failure | Alert at 21 days, correct DNS/ACME path, test renewal; never weaken TLS to hide expiry |
|
|
|
|
| Import failure | Mark incomplete; keep normal startup independent; no scheduling |
|
|
|
|
|
|
|
|
| Frontend/font failure | Keep previous frontend; no deployment-side asset workaround |
|
|
|
|
## Security model and failure modes
|
|
|
|
| Worker/Redis loss | Preserve PostgreSQL; reconcile/recompute idempotently |
|
|
|
|
|
|
|
|
| Certificate failure | Production readiness fails; do not serve plaintext application traffic |
|
|
|
|
### STRIDE summary
|
|
|
|
| Backup/upload failure | Preserve all prior verified copies and alert/operator-block destructive work |
|
|
|
|
|
|
|
|
| Restore target mismatch | Abort before database mutation |
|
|
|
|
| Threat | Platform control |
|
|
|
|
| Disk/log exhaustion risk | Bounded logs/resources and stop deploy before critical free-space floor |
|
|
|
|
|---|---|
|
|
|
|
| Spoofed operator | Key-only restricted SSH, named release record, minimal Docker access |
|
|
|
|
| Spoofed deploy/operator | Protected environments, hardware-backed/MFA human accounts where available, pinned SSH host key, least-privilege machine identity, signed/checksummed release manifest |
|
|
|
|
| Secret disclosure | File permissions, log allowlists, no env in backup; rotate first if exposed |
|
|
|
|
| Tampered image/config/backup | Digest-pinned images, protected release history, checksums, private registry, encrypted authenticated backups, trusted-source restore only |
|
|
|
|
|
|
|
|
| Repudiated production action | Append-only release/restore records with actor, approval, exact revisions, timestamps, and outcome |
|
|
|
|
|
|
|
|
| Information disclosure | Private networks, minimal ports, file-mounted secrets, log allowlists, no env in backups, no public monitoring/admin routes |
|
|
|
|
|
|
|
|
| Denial of service | Nginx limits/timeouts, bounded pools/resources/PIDs/logs, application rate limits, queue concurrency and 60-second job timeout |
|
|
|
|
|
|
|
|
| Elevation of privilege | Non-root containers, dropped capabilities, `no-new-privileges`, separated networks/secrets, restricted deploy command and operator path |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Failure behavior
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Missing/malformed configuration or secret: fail validation before mutation and name only the missing key/path, never its value.
|
|
|
|
|
|
|
|
- Unhealthy PostgreSQL/Redis: do not start/route dependent candidate services; preserve the last healthy release.
|
|
|
|
|
|
|
|
- Migration failure: stop release; never start candidate API/worker or auto-run down migration.
|
|
|
|
|
|
|
|
- Import failure: isolate the incomplete import record; normal deploy remains independent.
|
|
|
|
|
|
|
|
- Frontend/font validation failure: reject candidate before public switch; serve previous frontend.
|
|
|
|
|
|
|
|
- Backend readiness failure: keep/restore previous backend and record candidate logs/health evidence.
|
|
|
|
|
|
|
|
- Worker unavailable: API reports typed section degradation; queue age alert fires; ready cached sections remain usable.
|
|
|
|
|
|
|
|
- Redis memory exhaustion: reject unsafe new work, alert, and preserve PostgreSQL; do not silently evict authoritative state because Redis has none.
|
|
|
|
|
|
|
|
- Monitoring outage: application continues, but release gate warns/fails if the 15-minute observation cannot be performed.
|
|
|
|
|
|
|
|
- Backup/upload failure: retain previous verified backups, alert, and block destructive migrations.
|
|
|
|
|
|
|
|
- Restore validation mismatch: abort before destination mutation.
|
|
|
|
|
|
|
|
- Certificate missing/expired: production readiness fails; do not expose plaintext application traffic.
|
|
|
|
|
|
|
|
- Registry/forge unavailable: current release continues; deploy does not fall back to local mutable builds.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Always / Ask / Never boundaries
|
|
|
|
## Always / Ask / Never boundaries
|
|
|
|
|
|
|
|
|
|
|
|
### Always
|
|
|
|
### Always
|
|
|
|
|
|
|
|
|
|
|
|
- Deploy immutable, scanned image digests recorded with all three repository SHAs.
|
|
|
|
- Verify exact server, paths, repository remotes/branches/cleanliness, and record all three commit hashes before build.
|
|
|
|
- Validate exact environment, target path, configuration, backup age, health, migration, and rollback manifest before mutation.
|
|
|
|
- Build from the recorded checkouts and record the resulting release.
|
|
|
|
- Keep Nginx as the only public entry point and use TLS for all external traffic.
|
|
|
|
- Keep Nginx as the only public application entry point and use valid HTTPS.
|
|
|
|
- Separate migrations, CSV import, credential generation, API startup, and worker startup.
|
|
|
|
- Separate migrations, API startup, worker startup, CSV import, and credential generation.
|
|
|
|
- Keep Asynq pinned to `v0.26.0` and explicitly treat it as pre-1.0; test queue/retry/recovery before any upgrade.
|
|
|
|
- Keep Asynq pinned to pre-1.0 `v0.26.0` until compatibility/recovery tests approve a change.
|
|
|
|
- Mount secrets with least privilege, keep them out of Git/logs/backups/images, and rotate after exposure.
|
|
|
|
- Bound resources, timeouts, retries, logs, and backup retention.
|
|
|
|
- Bound resources, timeouts, retries, logs, metrics cardinality, backup retention, and operator actions.
|
|
|
|
- Back up PostgreSQL encrypted off-host and rehearse restore.
|
|
|
|
- Back up PostgreSQL encrypted off-host, verify the artifact, and rehearse restores.
|
|
|
|
- Use the ordinary frontend build context for the tracked font/license.
|
|
|
|
- Preserve the current healthy release until candidate readiness and smoke checks pass.
|
|
|
|
|
|
|
|
- Consume approved font/license files only through the ordinary frontend build and immutable frontend artifact.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Ask first
|
|
|
|
### Ask first
|
|
|
|
|
|
|
|
|
|
|
|
- Expose any new port, hostname, admin/debug/docs/metrics route, or third-party telemetry endpoint.
|
|
|
|
- Expose a new port, hostname, admin/debug/docs/metrics route, or third-party endpoint.
|
|
|
|
- Change PostgreSQL/Redis major version, persistence policy, network boundary, SLO, RPO/RTO, retention, or resource limit materially.
|
|
|
|
- Change PostgreSQL/Redis major version, backup provider/encryption, retention, RPO/RTO, or restore behavior.
|
|
|
|
- Run a destructive/non-backward-compatible migration, production restore, import replacement, or secret rotation that causes downtime.
|
|
|
|
- Run a destructive/non-compatible migration, production restore, or replacement import.
|
|
|
|
- Change deploy approval/trigger behavior, registry/runner trust, backup provider/encryption scheme, TLS/HSTS scope, or SSH access model.
|
|
|
|
- Change the manual latest-`main` deployment rule, repository topology, deploy-user privilege, or TLS/HSTS scope.
|
|
|
|
- Add a scheduled service, new external integration, new persistent volume, or public CDN.
|
|
|
|
- Add Prometheus/Grafana before MVP is ready or start AI before the observability phase is accepted.
|
|
|
|
- Change frontend ownership/location of the licensed font/license inputs.
|
|
|
|
- Change frontend ownership/location of the licensed assets.
|
|
|
|
|
|
|
|
|
|
|
|
### Never
|
|
|
|
### Never
|
|
|
|
|
|
|
|
|
|
|
|
- Push from an agent session, force-reset user history, or edit backend/frontend repositories from this deployment task.
|
|
|
|
- Add a CI/CD pipeline, runner, automated deployment trigger, or private image registry without a new approved decision.
|
|
|
|
- Build production images from mutable server checkouts or deploy branch names/floating tags.
|
|
|
|
- Push from an agent session or rewrite/discard user history.
|
|
|
|
- Publish PostgreSQL, Redis, backend, worker, monitoring, or Docker daemon ports to the internet.
|
|
|
|
- Publish PostgreSQL, Redis, backend, worker, monitoring, or Docker daemon ports to the internet.
|
|
|
|
- Run migrations or CSV import inside API startup; schedule/watch/poll the challenge CSV.
|
|
|
|
- Use unconditional `git reset --hard` on a dirty/diverged server checkout.
|
|
|
|
- Back up or restore Redis as authoritative data, or include `.env`, secret files, TLS private keys, raw CSV, credentials, or font copies in database backups.
|
|
|
|
- Run migration or CSV import inside API startup; schedule/watch/poll the CSV.
|
|
|
|
- Log secrets, raw session/cookie values, payer-card keys, raw payment rows, database/Redis URLs, or high-cardinality identifiers as metric labels.
|
|
|
|
- Include `.env`, secrets, TLS keys, Redis, raw CSV, credentials, logs, or font/license files in database backup.
|
|
|
|
- Automatically run down migrations or overwrite production during restore without the complete confirmation guard.
|
|
|
|
- Log credentials, raw tokens/cookies, payer-card keys, raw payment rows, or secret connection values.
|
|
|
|
- Add private font injection, copy the font/license into deployment Git, or diverge from the approved ordinary frontend build input.
|
|
|
|
- Automatically run down migrations or restore over production without every confirmation guard.
|
|
|
|
- Disable failed health/security/CI gates merely to complete a release.
|
|
|
|
- Add another font delivery/injection path or copy the assets into deployment Git.
|
|
|
|
|
|
|
|
- Disable health/security/backup gates to finish a deployment.
|
|
|
|
|
|
|
|
|
|
|
|
## Measurable acceptance criteria
|
|
|
|
## Measurable acceptance criteria
|
|
|
|
|
|
|
|
|
|
|
|
### Topology and exposure
|
|
|
|
### Manual deployment and rollback
|
|
|
|
|
|
|
|
|
|
|
|
- `docker compose config` validates and every production image resolves to an immutable digest.
|
|
|
|
- All three nested repositories resolve to approved remotes, latest `origin/main`, clean working trees, and recorded commit hashes before build.
|
|
|
|
- An external port scan finds only approved SSH plus TCP 80/443; PostgreSQL, Redis, backend, worker, frontend, Prometheus, and Grafana are unreachable directly.
|
|
|
|
- An induced pull/build failure leaves the active stack unchanged.
|
|
|
|
- Network inspection proves frontend/Nginx cannot connect to PostgreSQL/Redis and only required paths exist.
|
|
|
|
- A release record maps the running stack to three Git hashes, local image ids, migration version, operator, backup, and health evidence without secrets.
|
|
|
|
- Container policy tests prove non-root/read-only/capability/resource/log constraints or document an approved exception.
|
|
|
|
- Selecting the prior release hashes, rebuilding, and running checks completes application rollback within 15 minutes under normal conditions.
|
|
|
|
|
|
|
|
- No CI/CD workflow, runner configuration, registry dependency, or automated deployment trigger exists.
|
|
|
|
|
|
|
|
|
|
|
|
### Routing and TLS
|
|
|
|
### Topology, routing, and lifecycle
|
|
|
|
|
|
|
|
|
|
|
|
- HTTP and alternate hosts redirect to exactly `https://zarinpal.amiirkhl.ir`; unknown hosts are rejected.
|
|
|
|
- External scanning finds only approved SSH plus TCP 80/443; all internal services are unreachable directly.
|
|
|
|
- TLS accepts only 1.2/1.3 with a valid renewable certificate; security headers pass documented checks.
|
|
|
|
- HTTP redirects to `https://zarinpal.amiirkhl.ir`; unknown hosts are rejected; TLS issuance and renewal pass.
|
|
|
|
- `/api/` reaches only backend; application routes reach frontend; no SPA fallback captures API errors.
|
|
|
|
- `/api/` reaches only backend, application routes reach frontend, and SSE events/heartbeats are delivered without buffering.
|
|
|
|
- The authenticated SSE stream delivers a ready event without buffering, emits heartbeats within 20 seconds, preserves replay ids, and survives a controlled Nginx reload/reconnect.
|
|
|
|
- Clean startup health-gates PostgreSQL/Redis, runs migration once, then starts API/worker/frontend/Nginx.
|
|
|
|
|
|
|
|
- Normal startup/deploy never runs import/admin. Identical explicit import does not duplicate business rows.
|
|
|
|
|
|
|
|
- Redis deletion causes safe recomputation without loss of PostgreSQL payment/session data.
|
|
|
|
|
|
|
|
- Required font URL returns 200 with correct type, while deployment Git contains no font/license copies.
|
|
|
|
|
|
|
|
|
|
|
|
### Lifecycle and data safety
|
|
|
|
### Security and recovery
|
|
|
|
|
|
|
|
|
|
|
|
- A clean deployment waits for PostgreSQL/Redis health, runs migration once, then starts ready API/worker/frontend services.
|
|
|
|
- Real secrets remain outside Git with restrictive permissions and do not appear in Compose output/logs.
|
|
|
|
- A migration failure prevents candidate API/worker startup and leaves the prior release recoverable.
|
|
|
|
- Docker logs are structured/bounded and an induced backend failure is traceable by request/release id without sensitive fields.
|
|
|
|
- Normal startup/deploy never runs the importer. An explicit identical import re-run is idempotent and produces no duplicate business rows.
|
|
|
|
- Daily backup contains only PostgreSQL archive plus non-secret manifest/checksums, is encrypted, remotely verified, and retained three local/seven remote.
|
|
|
|
- Redis deletion followed by restart produces safe analytics recomputation and no loss of PostgreSQL facts or sessions.
|
|
|
|
- No excluded `.env`, secret, Redis, TLS, CSV, credential, log, monitoring, font, or license asset appears in the backup.
|
|
|
|
- Graceful deployment stops new work and leaves no section attempt running beyond its 60-second bound.
|
|
|
|
- Clean restore completes within two hours and passes schema/count/tenant/login/cold-cache/worker/SSE validation.
|
|
|
|
|
|
|
|
- Every missing production restore confirmation factor aborts before mutation.
|
|
|
|
|
|
|
|
|
|
|
|
### CI/CD and rollback
|
|
|
|
### Phase ordering
|
|
|
|
|
|
|
|
|
|
|
|
- Backend/frontend PRs cannot deploy; protected revisions publish one immutable image each only after their documented gates.
|
|
|
|
- MVP can be operated through health, logs, release records, disk/certificate/backup checks without Prometheus/Grafana.
|
|
|
|
- Candidate events cannot mutate production directly; one serialized deployment workflow assembles all three revisions into a release manifest.
|
|
|
|
- Prometheus/Grafana/exporters/dashboards/alerts are implemented and accepted after MVP readiness and before any AI implementation begins.
|
|
|
|
- A deployed instance exposes/report its release id internally, and it matches the manifest's three SHAs/digests.
|
|
|
|
|
|
|
|
- An induced candidate readiness failure preserves or restores the previous release. An application-only rollback completes within 15 minutes.
|
|
|
|
|
|
|
|
- No production job receives CI test credentials, and deployment artifacts/logs contain no secrets.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Font ownership
|
|
|
|
## Remaining decisions before implementation approval
|
|
|
|
|
|
|
|
|
|
|
|
- Frontend CI and deployment smoke verify `/fonts/IRANYekanXVFaNumVF.woff2` (or the exact frontend-approved emitted URL) returns 200 with the correct font content type.
|
|
|
|
1. Confirm server provider, Ubuntu version, CPU/RAM/disk, public IP/DNS owner, and deployment username/path.
|
|
|
|
- The deployment repository contains neither font binary nor `FontLicense.txt`; the active frontend image digest maps to the frontend SHA that owns both files.
|
|
|
|
2. Approve exact PostgreSQL 18, Redis 7 Alpine, Nginx, Certbot, and base-image tags after compatibility/security checks.
|
|
|
|
- No private asset fetch/injection step exists in deployment CI/CD.
|
|
|
|
3. Approve initial CPU, memory, PID, database-pool, log-rotation, and disk thresholds for the selected server.
|
|
|
|
|
|
|
|
4. Confirm exact backend/frontend repository URLs, build contexts, commands, health endpoints, migration/import/admin commands, and SSE path.
|
|
|
|
|
|
|
|
5. Confirm S3-compatible endpoint/bucket/prefix, new ZarinPal credentials, encryption passphrase custody, and cron time.
|
|
|
|
|
|
|
|
6. Confirm whether Redis uses an AOF named volume for routine restart continuity or starts fully disposable; it remains excluded from backup either way.
|
|
|
|
|
|
|
|
7. Approve the exact deployment release-record location/format and the safe behavior for a dirty or divergent server checkout.
|
|
|
|
|
|
|
|
8. Confirm who may deploy/restore and where off-server copies of production configuration/secrets are escrowed.
|
|
|
|
|
|
|
|
9. Define the explicit “MVP ready” gate that starts the Prometheus/Grafana phase and the observability acceptance gate that permits AI work.
|
|
|
|
|
|
|
|
|
|
|
|
### Observability and recovery
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- An induced backend 5xx is findable by request id and release id using structured logs without exposing sensitive fields.
|
|
|
|
|
|
|
|
- RED/USE, worker/SSE, database/Redis, host, backup, and TLS metrics appear with bounded labels and the defined alerts are test-fired.
|
|
|
|
|
|
|
|
- A daily encrypted backup is uploaded, checksum-verified, retained as three local/seven remote artifacts, and no excluded secret/config/runtime asset appears in it.
|
|
|
|
|
|
|
|
- A clean-destination restore of a selected backup completes within two hours, passes application/tenant/analytics validation, and records achieved RPO/RTO.
|
|
|
|
|
|
|
|
- A production restore attempt without any required confirmation factor aborts before database mutation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Implementation verification commands (to be finalized)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
These command interfaces are specification targets, not implemented commands:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
|
|
|
Validate: docker compose --env-file <non-secret-release-env> config --quiet
|
|
|
|
|
|
|
|
Inspect: docker compose --env-file <non-secret-release-env> ps
|
|
|
|
|
|
|
|
Logs: docker compose --env-file <non-secret-release-env> logs --since <duration> <service>
|
|
|
|
|
|
|
|
Migrate: deployment command -> one-off migrate service at candidate backend digest
|
|
|
|
|
|
|
|
Import: deployment command -> one-off import service with exact read-only source path
|
|
|
|
|
|
|
|
Backup: deployment command -> encrypted PostgreSQL backup plus remote verification
|
|
|
|
|
|
|
|
Restore: deployment command -> preflight, guarded empty-destination restore, validation
|
|
|
|
|
|
|
|
Rollback: deployment command -> previous immutable release manifest
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Exact script names and flags are implementation decisions constrained by the operator safety contract above. They are not authorized by this document's draft status.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Remaining decisions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Confirm server OS/provider, CPU/RAM/disk, backup object-store provider, and whether object lock/versioning is available.
|
|
|
|
|
|
|
|
2. Confirm Gitea/Actions/runner/registry versions and choose the supported authenticated cross-repository candidate event mechanism.
|
|
|
|
|
|
|
|
3. Decide rootless Docker versus a dedicated rootful Engine deploy account; document the chosen low-port and privilege model.
|
|
|
|
|
|
|
|
4. Approve exact PostgreSQL 18, Redis 7, Nginx, certbot, Prometheus/Grafana/exporter patch versions and digests after compatibility/security checks.
|
|
|
|
|
|
|
|
5. Approve concrete CPU/memory/PID/pool/log/metrics-retention budgets after production-shaped load testing.
|
|
|
|
|
|
|
|
6. Approve the exact backend health endpoints, worker heartbeat contract, API/SSE paths, OpenAPI compatibility metadata, and dedicated smoke-test account.
|
|
|
|
|
|
|
|
7. Choose the encrypted backup format/key-management mechanism and production restore database-switch strategy.
|
|
|
|
|
|
|
|
8. Decide whether certificate renewal runs as a Compose service or a host systemd timer; both must preserve Nginx reload and private-key permissions.
|
|
|
|
|
|
|
|
9. Confirm whether one human production approval is available in the forge and who owns deploy, backup, restore, and incident roles during the hackathon.
|
|
|
|
|
|
|
|
## References
|
|
|
|
## References
|
|
|
|
|
|
|
|
|
|
|
|
- Docker Compose application model: <https://docs.docker.com/compose/intro/compose-application-model/>
|
|
|
|
- Docker Compose application model: <https://docs.docker.com/compose/intro/compose-application-model/>
|
|
|
|
- Docker Compose startup order and health conditions: <https://docs.docker.com/compose/how-tos/startup-order/>
|
|
|
|
- Docker Compose startup order and health conditions: <https://docs.docker.com/compose/how-tos/startup-order/>
|
|
|
|
- Docker logging driver configuration: <https://docs.docker.com/engine/logging/configure/>
|
|
|
|
- Docker logging configuration: <https://docs.docker.com/engine/logging/configure/>
|
|
|
|
- Docker rootless mode: <https://docs.docker.com/engine/security/rootless/>
|
|
|
|
- Nginx proxy buffering/timeouts: <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>
|
|
|
|
- Nginx proxy module, including buffering/timeouts: <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>
|
|
|
|
|
|
|
|
- Nginx TLS module: <https://nginx.org/en/docs/http/ngx_http_ssl_module.html>
|
|
|
|
- Nginx TLS module: <https://nginx.org/en/docs/http/ngx_http_ssl_module.html>
|
|
|
|
- PostgreSQL 18 backup and restore: <https://www.postgresql.org/docs/current/backup.html>
|
|
|
|
- PostgreSQL 18 backup/restore: <https://www.postgresql.org/docs/current/backup.html>
|
|
|
|
- PostgreSQL SQL dump/restore guidance: <https://www.postgresql.org/docs/current/backup-dump.html>
|
|
|
|
- PostgreSQL dump/restore guidance: <https://www.postgresql.org/docs/current/backup-dump.html>
|
|
|
|
- Gitea Actions overview: <https://docs.gitea.com/usage/actions/overview/>
|
|
|
|
- Local read-only Qlockify reference: `D:/Programing/Scripts/Qlockify/qlockify-deployment`
|
|
|
|
- Gitea runner documentation: <https://docs.gitea.com/runner/>
|
|
|
|
|
|
|
|
- Local read-only reference: `D:/Programing/Scripts/Qlockify/qlockify-deployment`
|
|
|
|
|
|
|
|
|
