Files
zarinpal-analytics-deployment/docs/PLATFORM-OPERATIONS-SPEC.md

32 KiB

Platform Operations Capability Spec

  • Status: Draft revision — confirmed intent, ready for detailed stakeholder review
  • Implementation authorization: Not granted
  • Capability: Platform operations
  • Depends on: all deployable backend and frontend capability contracts
  • Last updated: 2026-08-20

Confirmed intent

Deploy zarinpal.amiirkhl.ir on one Ubuntu server using the proven Qlockify operating model: three nested Git repositories, Docker Compose builds performed on the server, Nginx as the only public entry point, automatic Let's Encrypt TLS, PostgreSQL 18, Redis 7 Alpine, a Go API and event-driven worker, and a static frontend.

Production deployment is manual. The operator pulls the latest main from all three repositories, records the resulting commit hashes, builds locally on the server, runs migrations and health checks, and records the release. Rollback checks out the previously recorded hashes and rebuilds. There is no CI/CD pipeline, runner, private image registry, or cross-repository automation.

MVP includes health checks, structured and bounded logs, safe startup, explicit migration/import separation, rollback, and encrypted daily PostgreSQL backup/restore adapted with minimal changes from Qlockify. Prometheus and Grafana are a required follow-up phase after the core project is ready and before AI work begins.

Objective

Provide a production-shaped hackathon platform that a trusted operator can understand and recover without hidden automation. A clean server can be prepared, the exact source revisions can be deployed, only approved traffic is publicly exposed, database changes and one-time data import are deliberate, failures leave useful evidence, and PostgreSQL can be restored from an encrypted off-host backup.

Scope

MVP scope

  • Qlockify-style nested repository layout and manual server-side deployment
  • Docker Compose service topology and internal networks
  • Nginx canonical routing, security headers, TLS termination, and SSE behavior
  • PostgreSQL 18, Redis 7 Alpine, backend API, frontend, and worker boundaries
  • Liveness/readiness, startup ordering, migration, shutdown, and resource limits
  • Explicit one-time CSV import and demo-credential administrative jobs
  • Environment, secret, volume, and host exposure policy
  • Manual release records, health verification, and Git-hash rollback
  • Structured container logs, log rotation, Docker health status, and operator checks
  • Daily encrypted PostgreSQL backup, S3-compatible upload, retention, and guarded restore adapted from Qlockify
  • Safe first-server bootstrap
  • Normal frontend build consumption of the tracked licensed font assets

Required post-MVP, pre-AI phase

  • Prometheus metrics collection
  • Grafana dashboards
  • Private exporters for host/container/PostgreSQL/Redis signals
  • Symptom-based alerts and alert delivery testing

This observability phase must be complete before the AI capability starts. It does not block the first working MVP deployment.

Out of scope

  • CI/CD pipelines, hosted/self-hosted runners, automated deploy triggers, and private image registries
  • Kubernetes, multi-server failover, multi-region operation, or zero-downtime claims
  • Scheduled or watched CSV ingestion
  • Redis backup or treating Redis as a source of truth
  • Public database, Redis, Docker, worker, metrics, debug, or administration endpoints
  • Backend/frontend application implementation in this repository
  • Private font injection or font copies in the deployment repository
  • AI implementation

Approved assumptions and decisions

  1. The target is one supported Ubuntu server.
  2. A dedicated trusted Ubuntu deployment user belongs to the docker group and runs Docker Compose. Docker-group membership is root-equivalent and is restricted to trusted operators.
  3. The deployment repository contains ignored nested checkouts of the backend and frontend repositories, matching ADR 0001 and Qlockify.
  4. Normal deployment selects the latest protected main from all three repositories, then records the exact three commit hashes before building.
  5. Application images are built on the production server from those recorded checkouts. They are not published to a registry.
  6. Rollback checks out the previous recorded hashes and rebuilds. Database rollback remains constrained by migration compatibility.
  7. Nginx obtains and renews a Let's Encrypt certificate automatically through Certbot using HTTP-01.
  8. PostgreSQL 18 and Redis 7 Alpine are the initial major versions. Exact patch tags are pinned before implementation.
  9. Redis is disposable cache/queue/coordination state and is excluded from backup. PostgreSQL remains authoritative.
  10. Daily encrypted PostgreSQL backups and tested restores are MVP requirements.
  11. Qlockify backup/restore/upload scripts are the implementation reference and will receive only necessary project path, service, credential, and safety changes.
  12. Production .env files, secrets, TLS keys, Redis data, raw CSV, credentials, logs, and font assets are excluded from database backup packages.
  13. Prometheus/Grafana are required after MVP readiness and before AI implementation.
  14. Frontend commit 64f5ac2 owns static/fonts/IRANYekanXVFaNumVF.woff2 and static/fonts/FontLicense.txt; deployment uses the normal frontend build context without another asset path.

Any change to these decisions requires a spec revision before implementation.

Repository and server layout

/home/<deploy-user>/zarinpal-analytics-deployment/
├── .git/                         deployment repository
├── backend/                      ignored checkout of backend repository
│   └── .git/
├── frontend/                     ignored checkout of frontend repository
│   └── .git/
├── docs/
├── backups/                      ignored runtime output
└── deployment assets             only after spec approval

Each checkout keeps its own Git history and remote. Deployment operations verify the expected repository, branch, remote, clean state, and resolved absolute path before pulling, checking out, or building.

Production configuration and secrets live outside public Git. Runtime volumes, backups, certificate data, raw CSV, and generated credentials remain ignored.

Runtime architecture

Internet
   |
   | TCP 80/443 only
   v
Nginx + Certbot
   |
   +------------------------+
   | private app network    |
   v                        v
frontend static server   backend API
                              |
                    +---------+---------+
                    | private data net |
                    v                  v
                PostgreSQL <------- Redis
                    ^                  ^
                    |                  |
              one-off jobs          worker
          migrate / import / admin

Nginx is the only public application entry point. The worker is event-driven and never scheduled. It uses Asynq pinned to v0.26.0, explicitly a pre-1.0 dependency. Redis loss causes controlled job/cache reconciliation and recomputation, not loss of authoritative payment or session data.

Docker Compose topology

Service Responsibility Networks State Host exposure
nginx TLS, canonical routing, limits, static/API/SSE proxy edge, app certificate mounts only 80/443
certbot Initial issuance and renewal edge ACME/certificate volume none
frontend Serve static SvelteKit build app none none
backend Go API, sessions, authorization, analytics control app, data none none
worker On-demand Asynq analytics sections data none none
postgres Authoritative database data postgres_data none
redis Queue, cache, rate-limit, replay coordination data optional routine AOF volume, excluded from backup none
migrate Versioned schema migration data none one-off only
import One-time CSV bootstrap/recovery data exact read-only input bind one-off profile only
admin Demo credentials and approved maintenance data explicit output bind if requested one-off profile only

Post-MVP adds prometheus, grafana, and approved exporters on a separate private monitoring network. Grafana binds only to loopback/private operator access unless a later ADR approves a protected public route.

Image and build policy

  • PostgreSQL, Redis, Nginx, Certbot, and later monitoring images use explicit supported tags; no latest tag.
  • Backend and worker build from the same recorded backend commit and deployment Dockerfile/context contract.
  • Frontend builds from the recorded frontend commit and normal build context, including its tracked font/license inputs.
  • The deployment script records resulting local image identifiers and build timestamps for diagnosis, but Git commit hashes are the reproducible source reference.
  • Build failure stops before replacing healthy running services.

Network and container policy

  • Only Nginx publishes ports 80 and 443.
  • PostgreSQL, Redis, backend, frontend, worker, and later monitoring services do not publish internet-facing ports.
  • Frontend and Nginx cannot connect directly to PostgreSQL or Redis.
  • Containers run as non-root where their official/application image supports it, drop unnecessary capabilities, set no-new-privileges, and use read-only filesystems where practical.
  • CPU, memory, PID, database-pool, and log limits are explicit. Initial values are validated against the actual server and import/load tests.
  • Docker logs use bounded rotation so a noisy service cannot fill the server disk.

Manual deployment and release records

Normal deployment

  1. Confirm exact server, deployment root, environment, available disk, recent backup, Docker health, and operator identity.
  2. Acquire a deployment lock so backup/restore/import/deploy operations cannot conflict unsafely.
  3. Verify all three repositories exist at their exact expected paths, use approved remotes, and contain no unexplained local changes.
  4. Fetch and fast-forward each repository to its latest origin/main. Do not discard local/user changes automatically.
  5. Record the deployment, backend, and frontend commit hashes before building.
  6. Validate required configuration/secret files without printing values.
  7. Render and validate Docker Compose configuration.
  8. Build candidate application/Nginx images on the server.
  9. Start/verify PostgreSQL and Redis.
  10. Run the dedicated migration service once and require success.
  11. Replace backend/worker, then frontend, then validate/reload Nginx.
  12. Run public and private health/smoke checks and observe for 15 minutes.
  13. Write a non-secret release record and mark it active only after checks pass.

The future deployment script adapts Qlockify's manual deploy.sh structure but must not use unconditional git reset --hard; a dirty or divergent checkout stops and asks the operator to resolve it.

Release record

Each server-side release record contains:

  • release id and UTC timestamp
  • deployment, backend, and frontend Git commit hashes
  • Git branch and remote identity for each checkout
  • resulting local image ids/tags
  • migration version
  • non-secret configuration checksum
  • operator identity
  • preflight backup id/age
  • pre/post health results
  • previous release id
  • outcome and rollback reason if applicable

It contains no secret values, .env content, raw CSV path/content, credentials, license text, font copy, database rows, or TLS private material.

Rollback

  • Select the previous known-good release record explicitly.
  • Verify the three target hashes exist, repository paths/remotes are correct, and working trees are clean.
  • Check out the previous deployment/backend/frontend hashes and rebuild locally.
  • Do not automatically reverse database migrations. Normal migrations use expand/contract compatibility so the previous application can run against the expanded schema.
  • Run the same health/smoke checks and record rollback as a new operational event.
  • Target application-only rollback time is 15 minutes under normal Git/build conditions.

Startup, migration, import, and shutdown

Startup order

  1. PostgreSQL and Redis start and become healthy.
  2. The one-off migration completes successfully.
  3. Backend and worker start and become ready.
  4. Frontend becomes healthy.
  5. Nginx routes production traffic only when required upstreams are ready.

Compose health/dependency conditions prevent common startup races, but the manual deployment operation still checks every step explicitly.

Migration contract

  • Application startup never performs migrations or automatic schema creation.
  • Migrations run from the recorded candidate backend revision under one migration lock.
  • Changes are backward-compatible through expand/deploy/contract sequencing.
  • Destructive or long-locking migrations require explicit approval, a fresh verified backup, measured evidence on production-shaped data, and a separate recovery plan.
  • Migration failure stops deployment before candidate backend/worker replacement.
  • Database down-migration is never automatic.

One-time CSV import

  • Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
  • The operator provides an exact resolved path outside Git; only that file is mounted read-only.
  • Import cannot overlap migration, restore, or another import.
  • The backend importer validates checksum/schema, streams data, reconciles counts, and records incomplete failure safely according to the Data Foundation spec.
  • An identical completed import is idempotent and does not duplicate business rows.
  • Demo credential generation is a separate one-time admin job whose plaintext output never enters Compose logs, Git, backups, or shell history by default.

Graceful shutdown

  • Nginx stops accepting new traffic before backend termination.
  • Backend drains within a bounded timeout.
  • Worker stops fetching new jobs; active attempts remain bounded by their existing 60-second timeout and retry rules.
  • PostgreSQL stops last. Forced termination is recorded as a deployment failure.

Health and readiness

Health responses are generic and never reveal versions, addresses, schema details, credentials, tenant data, or stack traces.

Component Liveness Readiness
Nginx process/config responds TLS virtual host can serve frontend and proxy backend
Frontend local static sentinel index and required built assets/font are readable
Backend process responds without dependency calls expected migration, PostgreSQL, and Redis capabilities are available
Worker process probe PostgreSQL/Redis available, queue consumer registered, heartbeat current
PostgreSQL process authenticated pg_isready for configured database/user
Redis process authenticated PING and required queue/cache commands

Initial health timing target is 10-second interval, 3-second timeout, 30-second start period, and five failures before unhealthy, revised only from measured startup evidence. Dependency failure changes readiness but does not deliberately create liveness restart storms.

Nginx, domain, SSE, and TLS

Routing

  • Port 80 serves /.well-known/acme-challenge/ and redirects everything else to https://zarinpal.amiirkhl.ir$request_uri.
  • Alternate approved hostnames redirect to the canonical hostname. Unknown hosts are rejected by a default server.
  • /api/ proxies only to the backend with host, scheme, safe client-address chain, and request id.
  • / serves/proxies the frontend; SPA fallback never captures API errors.
  • Fingerprinted assets receive long immutable caching; HTML/mutable manifests revalidate.
  • Public metrics, debug, profiling, database, Redis, worker, Docker, or administration routes do not exist.

SSE

  • The approved exact analytics SSE route gets a dedicated location.
  • Upstream HTTP/1.1 is used; Cookie, Last-Event-ID, request id, host, scheme, and client-forwarding headers are preserved.
  • Proxy buffering, caching, compression, and transformations are disabled for the route.
  • Read timeout is safely above the 15-second heartbeat; initial target is 75 seconds.
  • Every connection/replay is authenticated by the backend. Nginx does not bypass session or tenant checks.

TLS

  • Certbot obtains the certificate through HTTP-01 after DNS and port checks.
  • Staging issuance/renewal is tested before production issuance.
  • Nginx may temporarily use a self-signed certificate only during isolated first bootstrap; it is never production-ready.
  • TLS 1.2/1.3 only, with reviewed security headers.
  • HSTS is enabled only after production issuance and renewal are proven. Preload requires separate approval.
  • Certificate validity/expiry is checked daily in MVP operations; Prometheus alerting is added in the required post-MVP observability phase.

Configuration and secrets

  • A committed safe example names required variables and contains placeholders only.
  • Real production .env and secret files live outside public Git with mode 0600 or stricter.
  • Secrets are mounted only into the services that need them and never passed in command-line arguments or embedded in images.
  • Frontend receives no production secret.
  • Compose validation and diagnostics must not print resolved secret values.
  • Backup object-store credentials and encryption passphrase are read from protected server configuration, following the Qlockify script interface with renamed ZarinPal variables/paths as needed.
  • If a secret reaches Git/logs, rotate/revoke it first, then clean history/evidence as a separate incident action.

Licensed font

Frontend commit 64f5ac2 tracks static/fonts/IRANYekanXVFaNumVF.woff2 and static/fonts/FontLicense.txt. The normal frontend server build includes them. Deployment adds no fetch, injection, credential, duplicate binary, or license copy. Smoke checks verify the emitted font resource and content type without reproducing license text.

MVP logging and operator checks

Logs

  • Backend and worker emit structured JSON to stdout with UTC timestamp, level, service, release id, request id, analytics run/section/job ids where applicable, safe error class, and duration.
  • Nginx access logs include request id, safe route, status, bytes, request/upstream duration, and upstream status.
  • Never log passwords, hashes, cookies, raw tokens, payer-card keys, raw CSV rows, database/Redis URLs, backup keys, .env contents, or full merchant payloads.
  • Docker log rotation is explicit; initial target is 10 MiB per file and three files per container, adjusted from incident/disk evidence.
  • Production debug logging is off by default and time-bounded when enabled.

MVP manual checks

The operator can check:

  • Compose health and restart counts
  • Nginx/backend/worker/PostgreSQL/Redis logs
  • public TLS/canonical redirect and generic health
  • disk space/inodes and database/backup size
  • Redis memory and queue health
  • certificate days remaining and renewal result
  • last verified backup age and retention count
  • current/previous release records

An operator checklist/runbook provides exact commands after implementation approval.

Required post-MVP observability phase

Before AI implementation begins, add private Prometheus/Grafana and approved exporters/instrumentation for:

  • HTTP RED metrics with bounded route/status labels and latency histograms
  • worker queue depth/oldest age, section duration, retries/failures, concurrency, cache hit/miss, and SSE connection/replay behavior
  • PostgreSQL availability, connections, locks, query duration aggregates, database/volume size
  • Redis availability, memory, eviction, persistence, connections, and queue state
  • host/container CPU, memory, filesystem/inodes, OOM/restarts, and network
  • backup success/age/duration/size/remote verification and restore-rehearsal age
  • TLS expiry and renewal status

Grafana and Prometheus remain private. Alerts are symptom-based, actionable, link to runbooks, and use only page/ticket severity. Merchant/user/request/card ids never become metric labels.

Initial targets to validate:

Objective Target
Public availability 99.5% over rolling 7 days excluding approved maintenance
Unexpected API 5xx under 1% over 5 minutes under acceptance traffic
Non-analytics API latency p95 under 500 ms under acceptance load
Analytics start p95 under 1 second
SSE connection p95 under 2 seconds; healthy heartbeat gap at most 20 seconds
Section jobs queue delay p95 under 5 seconds; every attempt at most 60 seconds
Capacity at least 20% and 10 GiB free on critical filesystems
Backup verified encrypted off-host backup every 24 hours
Restore clean rehearsal within two hours
TLS valid automatically renewable certificate

Backup and restore

Qlockify adaptation boundary

Reuse and minimally adapt these read-only reference scripts after implementation approval:

  • qlockify-deployment/scripts/backup.sh
  • qlockify-deployment/scripts/backup-upload-s3.sh
  • qlockify-deployment/scripts/restore.sh
  • qlockify-deployment/scripts/restore-from-s3.sh

Necessary changes include deployment root, archive/prefix names, Compose service names, protected credentials, PostgreSQL-only contents, manifest/checksums, exact target validation, and production confirmation guards. The core pg_dump/archive, OpenSSL encryption, rclone S3-compatible upload, local/remote retention, download/decrypt, and restore flow remains recognizable.

Backup contract

  • Daily PostgreSQL logical dump using a matching PostgreSQL 18 client.
  • Backup package contains database archive plus non-secret manifest/checksums only.
  • Encrypt before durable local/remote storage using the approved Qlockify-compatible mechanism.
  • Upload to private S3-compatible storage over TLS and verify the remote object.
  • Retain the latest three verified encrypted local backups and seven verified encrypted remote backups.
  • Delete stale backups only after a new remote backup is verified; never delete the last known-good copy.
  • Exclude .env, secrets, Redis, TLS keys, raw CSV, credentials, logs, monitoring state, font binary, and FontLicense.txt.
  • A dump is successful only after archive listing/checksum, encryption, upload, remote verification, and manifest completion.

Restore contract

  • Default restore target is a new empty non-production database.
  • Validate explicit backup id, checksums, manifest, archive listing, PostgreSQL compatibility, free space, and exact resolved destination before mutation.
  • Production restore additionally requires environment, host, database, backup id, typed confirmation, maintenance mode, exclusive lock, and named operator.
  • Never pipe an unchecked dump into the active database or restore over a running destination in place.
  • Restore into an empty database/cluster, validate counts/invariants/tenant isolation/login/cold Redis analytics/SSE, then switch/reopen traffic.
  • Rehearse monthly and before destructive migrations; record achieved RPO/RTO.

Safe first-server bootstrap

  1. Record provider/instance, OS, network, capacity, recovery console, operators, DNS, and rollback window.
  2. Prove two key-only operator logins and provider console recovery before disabling root/password SSH.
  3. Configure default-deny firewall: operator-allowlisted SSH plus public 80/443 only.
  4. Patch the OS, enable time sync/security updates/log rotation/disk monitoring, and reboot-test access.
  5. Install Docker Engine/Compose from the approved source and add only the dedicated trusted deploy user to the docker group.
  6. Clone the deployment repository and backend/frontend nested repositories into exact validated paths.
  7. Create protected configuration/secret paths outside Git and validate permissions.
  8. Validate Compose and local builds without starting public traffic.
  9. Point DNS, test ACME staging, issue production TLS, and verify canonical routing.
  10. Start data services, run migration, then API/worker/frontend/Nginx and health checks.
  11. Run the explicit one-time CSV import and separate credential generation only after the platform is healthy.
  12. Complete encrypted remote backup and clean restore rehearsal before handoff.

Threats and failure behavior

Failure/threat Required behavior
Wrong server/path/repository Stop before Git, build, volume, or database mutation
Dirty/diverged checkout Stop and ask operator; never unconditional hard reset
Pull/build failure Keep current containers/release; record evidence
Missing/malformed secret/config Fail before mutation without printing value
PostgreSQL/Redis unhealthy Do not start/route dependent candidate services
Migration failure Stop release; no candidate backend/worker; no automatic down migration
Import failure Mark incomplete; keep normal startup independent; no scheduling
Frontend/font failure Keep previous frontend; no deployment-side asset workaround
Worker/Redis loss Preserve PostgreSQL; reconcile/recompute idempotently
Certificate failure Production readiness fails; do not serve plaintext application traffic
Backup/upload failure Preserve all prior verified copies and alert/operator-block destructive work
Restore target mismatch Abort before database mutation
Disk/log exhaustion risk Bounded logs/resources and stop deploy before critical free-space floor
Spoofed operator Key-only restricted SSH, named release record, minimal Docker access
Secret disclosure File permissions, log allowlists, no env in backup; rotate first if exposed

Always / Ask / Never boundaries

Always

  • Verify exact server, paths, repository remotes/branches/cleanliness, and record all three commit hashes before build.
  • Build from the recorded checkouts and record the resulting release.
  • Keep Nginx as the only public application entry point and use valid HTTPS.
  • Separate migrations, API startup, worker startup, CSV import, and credential generation.
  • Keep Asynq pinned to pre-1.0 v0.26.0 until compatibility/recovery tests approve a change.
  • Bound resources, timeouts, retries, logs, and backup retention.
  • Back up PostgreSQL encrypted off-host and rehearse restore.
  • Use the ordinary frontend build context for the tracked font/license.

Ask first

  • Expose a new port, hostname, admin/debug/docs/metrics route, or third-party endpoint.
  • Change PostgreSQL/Redis major version, backup provider/encryption, retention, RPO/RTO, or restore behavior.
  • Run a destructive/non-compatible migration, production restore, or replacement import.
  • Change the manual latest-main deployment rule, repository topology, deploy-user privilege, or TLS/HSTS scope.
  • Add Prometheus/Grafana before MVP is ready or start AI before the observability phase is accepted.
  • Change frontend ownership/location of the licensed assets.

Never

  • Add a CI/CD pipeline, runner, automated deployment trigger, or private image registry without a new approved decision.
  • Push from an agent session or rewrite/discard user history.
  • Publish PostgreSQL, Redis, backend, worker, monitoring, or Docker daemon ports to the internet.
  • Use unconditional git reset --hard on a dirty/diverged server checkout.
  • Run migration or CSV import inside API startup; schedule/watch/poll the CSV.
  • Include .env, secrets, TLS keys, Redis, raw CSV, credentials, logs, or font/license files in database backup.
  • Log credentials, raw tokens/cookies, payer-card keys, raw payment rows, or secret connection values.
  • Automatically run down migrations or restore over production without every confirmation guard.
  • Add another font delivery/injection path or copy the assets into deployment Git.
  • Disable health/security/backup gates to finish a deployment.

Measurable acceptance criteria

Manual deployment and rollback

  • All three nested repositories resolve to approved remotes, latest origin/main, clean working trees, and recorded commit hashes before build.
  • An induced pull/build failure leaves the active stack unchanged.
  • A release record maps the running stack to three Git hashes, local image ids, migration version, operator, backup, and health evidence without secrets.
  • Selecting the prior release hashes, rebuilding, and running checks completes application rollback within 15 minutes under normal conditions.
  • No CI/CD workflow, runner configuration, registry dependency, or automated deployment trigger exists.

Topology, routing, and lifecycle

  • External scanning finds only approved SSH plus TCP 80/443; all internal services are unreachable directly.
  • HTTP redirects to https://zarinpal.amiirkhl.ir; unknown hosts are rejected; TLS issuance and renewal pass.
  • /api/ reaches only backend, application routes reach frontend, and SSE events/heartbeats are delivered without buffering.
  • Clean startup health-gates PostgreSQL/Redis, runs migration once, then starts API/worker/frontend/Nginx.
  • Normal startup/deploy never runs import/admin. Identical explicit import does not duplicate business rows.
  • Redis deletion causes safe recomputation without loss of PostgreSQL payment/session data.
  • Required font URL returns 200 with correct type, while deployment Git contains no font/license copies.

Security and recovery

  • Real secrets remain outside Git with restrictive permissions and do not appear in Compose output/logs.
  • Docker logs are structured/bounded and an induced backend failure is traceable by request/release id without sensitive fields.
  • Daily backup contains only PostgreSQL archive plus non-secret manifest/checksums, is encrypted, remotely verified, and retained three local/seven remote.
  • No excluded .env, secret, Redis, TLS, CSV, credential, log, monitoring, font, or license asset appears in the backup.
  • Clean restore completes within two hours and passes schema/count/tenant/login/cold-cache/worker/SSE validation.
  • Every missing production restore confirmation factor aborts before mutation.

Phase ordering

  • MVP can be operated through health, logs, release records, disk/certificate/backup checks without Prometheus/Grafana.
  • Prometheus/Grafana/exporters/dashboards/alerts are implemented and accepted after MVP readiness and before any AI implementation begins.

Remaining decisions before implementation approval

  1. Confirm server provider, Ubuntu version, CPU/RAM/disk, public IP/DNS owner, and deployment username/path.
  2. Approve exact PostgreSQL 18, Redis 7 Alpine, Nginx, Certbot, and base-image tags after compatibility/security checks.
  3. Approve initial CPU, memory, PID, database-pool, log-rotation, and disk thresholds for the selected server.
  4. Confirm exact backend/frontend repository URLs, build contexts, commands, health endpoints, migration/import/admin commands, and SSE path.
  5. Confirm S3-compatible endpoint/bucket/prefix, new ZarinPal credentials, encryption passphrase custody, and cron time.
  6. Confirm whether Redis uses an AOF named volume for routine restart continuity or starts fully disposable; it remains excluded from backup either way.
  7. Approve the exact deployment release-record location/format and the safe behavior for a dirty or divergent server checkout.
  8. Confirm who may deploy/restore and where off-server copies of production configuration/secrets are escrowed.
  9. Define the explicit “MVP ready” gate that starts the Prometheus/Grafana phase and the observability acceptance gate that permits AI work.

References