starcoach-hackathon/zarinpal-analytics-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
zarinpal-analytics-deployment/docs/PLATFORM-OPERATIONS-SPEC.md

35 KiB
Raw Permalink Blame History

Platform Operations Capability Spec

  • Status: Approved specification
  • Implementation authorization: Granted by user on 2026-08-20
  • Capability: Platform operations
  • Depends on: all deployable backend and frontend capability contracts
  • Last updated: 2026-08-20

2026-08-22 ingress amendment: ADR 0006 supersedes the dedicated Nginx/Certbot edge described below. The host's existing Caddy deployment owns ports 80/443, TLS, canonical-host routing, security headers, API/SSE proxying, and frontend proxying. This project publishes no ports and attaches only backend and frontend to the external caddy_proxy network. All private-data, manual-release, migration/import, secret, health, backup, and recovery requirements remain in force.

Confirmed intent

Deploy zarinpal.amiirkhl.ir on one Ubuntu server using the proven Qlockify operating model: three nested Git repositories, Docker Compose builds performed on the server, Nginx as the only public entry point, automatic Let's Encrypt TLS, PostgreSQL 18, Redis 7 Alpine, a Go API and event-driven worker, and a static frontend.

Production deployment is manual. The operator pulls the latest main from all three repositories, records the resulting commit hashes, builds locally on the server, runs migrations and health checks, and records the release. Rollback checks out the previously recorded hashes and rebuilds. There is no CI/CD pipeline, runner, private image registry, or cross-repository automation.

MVP includes health checks, structured and bounded logs, safe startup, explicit migration/import separation, rollback, and encrypted daily PostgreSQL backup/restore adapted with minimal changes from Qlockify. Prometheus and Grafana are a required follow-up phase after the core project is ready and before AI work begins.

Objective

Provide a production-shaped hackathon platform that a trusted operator can understand and recover without hidden automation. A clean server can be prepared, the exact source revisions can be deployed, only approved traffic is publicly exposed, database changes and one-time data import are deliberate, failures leave useful evidence, and PostgreSQL can be restored from an encrypted off-host backup.

Scope

MVP scope

  • Qlockify-style nested repository layout and manual server-side deployment
  • Docker Compose service topology and internal networks
  • Nginx canonical routing, security headers, TLS termination, and SSE behavior
  • PostgreSQL 18, Redis 7 Alpine, backend API, frontend, and worker boundaries
  • Liveness/readiness, startup ordering, migration, shutdown, and resource limits
  • Explicit one-time CSV import and demo-credential administrative jobs
  • Environment, secret, volume, and host exposure policy
  • Manual release records, health verification, and Git-hash rollback
  • Structured container logs, log rotation, Docker health status, and operator checks
  • Daily encrypted PostgreSQL backup, S3-compatible upload, retention, and guarded restore adapted from Qlockify
  • Safe first-server bootstrap
  • Normal frontend build consumption of the tracked licensed font assets

Required post-MVP, pre-AI phase

  • Prometheus metrics collection
  • Grafana dashboards
  • Private exporters for host/container/PostgreSQL/Redis signals
  • Symptom-based alerts and alert delivery testing

This observability phase must be complete before the AI capability starts. It does not block the first working MVP deployment.

Out of scope

  • CI/CD pipelines, hosted/self-hosted runners, automated deploy triggers, and private image registries
  • Kubernetes, multi-server failover, multi-region operation, or zero-downtime claims
  • Scheduled or watched CSV ingestion
  • Redis backup or treating Redis as a source of truth
  • Public database, Redis, Docker, worker, metrics, debug, or administration endpoints
  • Backend/frontend application implementation in this repository
  • Private font injection or font copies in the deployment repository
  • AI implementation

Approved assumptions and decisions

  1. The target is a new dedicated Ubuntu 24.04 LTS server with 2 CPU cores, 2 GB RAM, Docker already installed, user ubuntu, and a bootstrap-created 2 GB swap file.
  2. The trusted ubuntu user belongs to the docker group and runs Docker Compose. Docker-group membership is root-equivalent, so access to this account is restricted to trusted operators.
  3. The deployment repository contains ignored nested checkouts of the backend and frontend repositories, matching ADR 0001 and Qlockify.
  4. The operator manually runs git pull --ff-only origin main in all three repositories and enters HTTPS credentials interactively. The deployment command performs no pull or credential handling; it verifies and records the three commits before building.
  5. Application images are built on the production server from those recorded checkouts. They are not published to a registry.
  6. Rollback checks out the previous recorded hashes and rebuilds. Database rollback remains constrained by migration compatibility.
  7. Nginx obtains and renews a Let's Encrypt certificate automatically through Certbot using HTTP-01.
  8. Infrastructure images are pinned to postgres:18.4-alpine3.24, redis:7.4.10-alpine3.21, nginx:1.29.8-alpine3.23, and certbot/certbot:v5.7.0.
  9. Redis uses a named volume, AOF everysec, and noeviction for routine restart continuity. It remains disposable cache/queue/coordination state and is excluded from backup; PostgreSQL remains authoritative.
  10. Daily encrypted PostgreSQL backups and tested restores are MVP requirements.
  11. Qlockify backup/restore/upload scripts are the implementation reference and will receive only necessary project path, service, credential, and safety changes.
  12. Production .env files, secrets, TLS keys, Redis data, raw CSV, credentials, logs, and font assets are excluded from database backup packages.
  13. Prometheus/Grafana are required after MVP readiness and before AI implementation. MVP readiness requires application flows, tenant isolation, migration/import, TLS, manual health/log checks, rollback, encrypted backup, and clean restore to pass.
  14. Frontend commit 64f5ac2 owns static/fonts/IRANYekanXVFaNumVF.woff2 and static/fonts/FontLicense.txt; deployment uses the normal frontend build context without another asset path.

Any change to these decisions requires a spec revision before implementation.

Repository and server layout

/home/ubuntu/zarinpal-analytics-deployment/
├── .git/                         deployment repository
├── backend/                      ignored checkout of backend repository
│   └── .git/
├── frontend/                     ignored checkout of frontend repository
│   └── .git/
├── docs/
├── backups/                      ignored runtime output
└── deployment assets             only after explicit implementation authorization

Each checkout keeps its own Git history and remote. Deployment operations verify the expected repository, branch, remote, clean state, and resolved absolute path before pulling, checking out, or building.

Production configuration and secrets live outside public Git. Runtime volumes, backups, certificate data, raw CSV, and generated credentials remain ignored.

Runtime architecture

Internet
   |
   | TCP 80/443 only
   v
Nginx + Certbot
   |
   +------------------------+
   | private app network    |
   v                        v
frontend static server   backend API
                              |
                    +---------+---------+
                    | private data net |
                    v                  v
                PostgreSQL <------- Redis
                    ^                  ^
                    |                  |
              one-off jobs          worker
          migrate / import / admin

Nginx is the only public application entry point. The worker is event-driven and never scheduled. It uses Asynq pinned to v0.26.0, explicitly a pre-1.0 dependency. Redis loss causes controlled job/cache reconciliation and recomputation, not loss of authoritative payment or session data.

Docker Compose topology

Service Responsibility Networks State Host exposure
nginx TLS, canonical routing, limits, static/API/SSE proxy edge, app certificate mounts only 80/443
certbot Initial issuance and renewal edge ACME/certificate volume none
frontend Serve static SvelteKit build app none none
backend Go API, sessions, authorization, analytics control app, data none none
worker On-demand Asynq analytics sections data none none
postgres Authoritative database data postgres_data none
redis Queue, cache, rate-limit, replay coordination data named AOF volume, excluded from backup none
migrate Versioned schema migration data none one-off only
import One-time CSV bootstrap/recovery data exact read-only input bind one-off profile only
calendar Reviewed catalog bootstrap with audited state transitions data reviewed read-only JSON bind one-off profile only
admin Demo credentials and approved maintenance data explicit output bind if requested one-off profile only

Post-MVP adds prometheus, grafana, and approved exporters on a separate private monitoring network. Grafana binds only to loopback/private operator access unless a later ADR approves a protected public route.

Image and build policy

  • PostgreSQL, Redis, Nginx, and Certbot use the exact approved tags above; no latest tag.
  • Backend and worker build from the same recorded backend commit and deployment Dockerfile/context contract.
  • Frontend builds from the recorded frontend commit and normal build context, including its tracked font/license inputs.
  • The deployment script records resulting local image identifiers and build timestamps for diagnosis, but Git commit hashes are the reproducible source reference.
  • Build failure stops before replacing healthy running services.

Network and container policy

  • Only Nginx publishes ports 80 and 443.
  • PostgreSQL, Redis, backend, frontend, worker, and later monitoring services do not publish internet-facing ports.
  • Frontend and Nginx cannot connect directly to PostgreSQL or Redis.
  • Containers run as non-root where their official/application image supports it, drop unnecessary capabilities, set no-new-privileges, and use read-only filesystems where practical.
  • CPU, memory, PID, database-pool, and log limits are explicit. Initial values are validated against the actual server and import/load tests.
  • Docker logs use bounded rotation so a noisy service cannot fill the server disk.

Initial runtime limits for the 2-GB server are:

Service Memory limit
PostgreSQL 512 MB
Redis 256 MB container limit; 192 MB maxmemory target to leave process/AOF overhead
Backend API 256 MB
Worker 512 MB
Frontend 96 MB
Nginx 96 MB

Certbot is one-shot/periodic rather than continuously memory-resident. Worker concurrency remains 16 globally and 4 per merchant. If measured load exceeds this server, upgrade server capacity rather than silently lowering the approved concurrency.

Manual deployment and release records

Normal deployment

  1. Confirm exact server, deployment root, environment, available disk, recent backup, Docker health, and operator identity.
  2. Acquire a deployment lock so backup/restore/import/deploy operations cannot conflict unsafely.
  3. Before running the deployment command, the operator manually executes git pull --ff-only origin main in each repository and enters HTTPS credentials interactively; no Git credential is stored by deployment.
  4. Verify all three repositories exist at their exact expected paths, use the approved HTTPS remotes, are on main, match origin/main, and contain no unexplained local changes.
  5. Record the deployment, backend, and frontend commit hashes under the ignored releases/ directory before building.
  6. Validate required configuration/secret files without printing values.
  7. Render and validate Docker Compose configuration.
  8. Build candidate application/Nginx images on the server.
  9. Start/verify PostgreSQL and Redis.
  10. Run the dedicated migration service once and require success.
  11. Replace backend/worker, then frontend, then validate/reload Nginx.
  12. Run public and private health/smoke checks and observe for 15 minutes.
  13. Write a non-secret release record and mark it active only after checks pass.

The future deployment script adapts Qlockify's manual deploy.sh structure but must not use unconditional git reset --hard; a dirty or divergent checkout stops and asks the operator to resolve it.

Release record

Each server-side release record contains:

  • release id and UTC timestamp
  • deployment, backend, and frontend Git commit hashes
  • Git branch and remote identity for each checkout
  • resulting local image ids/tags
  • migration version
  • non-secret configuration checksum
  • operator identity
  • preflight backup id/age
  • pre/post health results
  • previous release id
  • outcome and rollback reason if applicable

It contains no secret values, .env content, raw CSV path/content, credentials, license text, font copy, database rows, or TLS private material.

Rollback

  • Select the previous known-good release record explicitly.
  • Verify the three target hashes exist, repository paths/remotes are correct, and working trees are clean.
  • Check out the previous deployment/backend/frontend hashes and rebuild locally.
  • Do not automatically reverse database migrations. Normal migrations use expand/contract compatibility so the previous application can run against the expanded schema.
  • Run the same health/smoke checks and record rollback as a new operational event.
  • Target application-only rollback time is 15 minutes under normal Git/build conditions.

Startup, migration, import, and shutdown

Startup order

  1. PostgreSQL and Redis start and become healthy.
  2. The one-off migration completes successfully.
  3. Backend and worker start and become ready.
  4. Frontend becomes healthy.
  5. Nginx routes production traffic only when required upstreams are ready.

Compose health/dependency conditions prevent common startup races, but the manual deployment operation still checks every step explicitly.

Migration contract

  • Application startup never performs migrations or automatic schema creation.
  • Migrations run from the recorded candidate backend revision under one migration lock.
  • Changes are backward-compatible through expand/deploy/contract sequencing.
  • Destructive or long-locking migrations require explicit approval, a fresh verified backup, measured evidence on production-shaped data, and a separate recovery plan.
  • Migration failure stops deployment before candidate backend/worker replacement.
  • Database down-migration is never automatic.

One-time CSV import

  • Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
  • On the first server release only, tracked scripts/first-start.sh orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, reviewed calendar bootstrap, and then public application startup. Later ordinary deploys/restarts never invoke it.
  • The operator provides an exact resolved path outside Git; only that file is mounted read-only.
  • Import cannot overlap restore, backup, or another import and starts only after migration succeeds.
  • The backend importer pins the approved checksum/schema/filename/counts, enforces resource caps, streams data, reconciles counts, removes temporary raw staging, and records incomplete failure safely according to the approved backend import specifications.
  • An identical completed import is idempotent and does not duplicate business rows.
  • Import failure prevents the first public application activation. Recovery restarts from the immutable archive; it never resumes guessed partial CSV/database state.
  • Demo credential generation is a separate one-time admin job whose plaintext output never enters Compose logs, Git, backups, or shell history by default.

Graceful shutdown

  • Nginx stops accepting new traffic before backend termination.
  • Backend drains within a bounded timeout.
  • Worker stops fetching new jobs; active attempts remain bounded by their existing 60-second timeout and retry rules.
  • PostgreSQL stops last. Forced termination is recorded as a deployment failure.

Health and readiness

Health responses are generic and never reveal versions, addresses, schema details, credentials, tenant data, or stack traces.

Component Liveness Readiness
Nginx process/config responds TLS virtual host can serve frontend and proxy backend
Frontend local static sentinel index and required built assets/font are readable
Backend process responds without dependency calls expected migration, PostgreSQL, and Redis capabilities are available
Worker process probe PostgreSQL/Redis available, queue consumer registered, heartbeat current
PostgreSQL process authenticated pg_isready for configured database/user
Redis process authenticated PING and required queue/cache commands

Initial health timing target is 10-second interval, 3-second timeout, 30-second start period, and five failures before unhealthy, revised only from measured startup evidence. Dependency failure changes readiness but does not deliberately create liveness restart storms.

Nginx, domain, SSE, and TLS

Routing

  • Port 80 serves /.well-known/acme-challenge/ and redirects everything else to https://zarinpal.amiirkhl.ir$request_uri.
  • The certificate and application serve only zarinpal.amiirkhl.ir; there is no www or alternate domain. Unknown hosts are rejected by a default server.
  • /api/ proxies only to the backend with host, scheme, safe client-address chain, and request id.
  • / serves/proxies the frontend; SPA fallback never captures API errors.
  • Fingerprinted assets receive long immutable caching; HTML/mutable manifests revalidate.
  • Public metrics, debug, profiling, database, Redis, worker, Docker, or administration routes do not exist.

SSE

  • The approved exact analytics SSE route gets a dedicated location.
  • Upstream HTTP/1.1 is used; Cookie, Last-Event-ID, request id, host, scheme, and client-forwarding headers are preserved.
  • Proxy buffering, caching, compression, and transformations are disabled for the route.
  • Read timeout is safely above the 15-second heartbeat; initial target is 75 seconds.
  • Every connection/replay is authenticated by the backend. Nginx does not bypass session or tenant checks.

TLS

  • Certbot obtains the certificate through HTTP-01 after DNS and port checks.
  • Staging issuance/renewal is tested before production issuance.
  • Nginx may temporarily use a self-signed certificate only during isolated first bootstrap; it is never production-ready.
  • TLS 1.2/1.3 only, with reviewed security headers.
  • HSTS is enabled only after production issuance and renewal are proven. Preload requires separate approval.
  • Certificate validity/expiry is checked daily in MVP operations; Prometheus alerting is added in the required post-MVP observability phase.

Configuration and secrets

  • A committed safe example names required variables and contains placeholders only.
  • Follow the Qlockify environment layout: deployment .env in the deployment root plus backend/frontend .env files in their nested repository roots. All are Git-ignored, mode 0600, and excluded from backups.
  • Secrets are mounted only into the services that need them and never passed in command-line arguments or embedded in images.
  • Frontend receives no production secret.
  • Compose validation and diagnostics must not print resolved secret values.
  • Backup object-store credentials and encryption passphrase are read from the protected deployment .env, following the Qlockify script interface with ZarinPal-specific ParsPack bucket/prefix and separate credentials.
  • If a secret reaches Git/logs, rotate/revoke it first, then clean history/evidence as a separate incident action.

Licensed font

Frontend commit 64f5ac2 tracks static/fonts/IRANYekanXVFaNumVF.woff2 and static/fonts/FontLicense.txt. The normal frontend server build includes them. Deployment adds no fetch, injection, credential, duplicate binary, or license copy. Smoke checks verify the emitted font resource and content type without reproducing license text.

MVP logging and operator checks

Logs

  • Backend and worker emit structured JSON to stdout with UTC timestamp, level, service, release id, request id, analytics run/section/job ids where applicable, safe error class, and duration.
  • Nginx access logs include request id, safe route, status, bytes, request/upstream duration, and upstream status.
  • Never log passwords, hashes, cookies, raw tokens, payer-card keys, raw CSV rows, database/Redis URLs, backup keys, .env contents, or full merchant payloads.
  • Docker log rotation is explicit; initial target is 10 MiB per file and three files per container, adjusted from incident/disk evidence.
  • Production debug logging is off by default and time-bounded when enabled.

MVP manual checks

The operator can check:

  • Compose health and restart counts
  • Nginx/backend/worker/PostgreSQL/Redis logs
  • public TLS/canonical redirect and generic health
  • disk space/inodes and database/backup size
  • Redis memory and queue health
  • certificate days remaining and renewal result
  • last verified backup age and retention count
  • current/previous release records

An operator checklist/runbook provides exact commands after implementation approval.

Required post-MVP observability phase

Before AI implementation begins, add private Prometheus/Grafana and approved exporters/instrumentation for:

  • HTTP RED metrics with bounded route/status labels and latency histograms
  • worker queue depth/oldest age, section duration, retries/failures, concurrency, cache hit/miss, and SSE connection/replay behavior
  • PostgreSQL availability, connections, locks, query duration aggregates, database/volume size
  • Redis availability, memory, eviction, persistence, connections, and queue state
  • host/container CPU, memory, filesystem/inodes, OOM/restarts, and network
  • backup success/age/duration/size/remote verification and restore-rehearsal age
  • TLS expiry and renewal status

Grafana and Prometheus remain private. Alerts are symptom-based, actionable, link to runbooks, and use only page/ticket severity. Merchant/user/request/card ids never become metric labels.

Initial targets to validate:

Objective Target
Public availability 99.5% over rolling 7 days excluding approved maintenance
Unexpected API 5xx under 1% over 5 minutes under acceptance traffic
Non-analytics API latency p95 under 500 ms under acceptance load
Analytics start p95 under 1 second
SSE connection p95 under 2 seconds; healthy heartbeat gap at most 20 seconds
Section jobs queue delay p95 under 5 seconds; every attempt at most 60 seconds
Capacity at least 20% and 10 GiB free on critical filesystems
Backup verified encrypted off-host backup every 24 hours
Restore clean rehearsal within two hours
TLS valid automatically renewable certificate

Backup and restore

Qlockify adaptation boundary

Reuse and minimally adapt these read-only reference scripts after implementation approval:

  • qlockify-deployment/scripts/backup.sh
  • qlockify-deployment/scripts/backup-upload-s3.sh
  • qlockify-deployment/scripts/restore.sh
  • qlockify-deployment/scripts/restore-from-s3.sh

Necessary changes include deployment root, archive/prefix names, Compose service names, protected credentials, PostgreSQL-only contents, manifest/checksums, exact target validation, and production confirmation guards. The core pg_dump/archive, OpenSSL encryption, rclone S3-compatible upload, local/remote retention, download/decrypt, and restore flow remains recognizable.

Backup contract

  • Daily at 02:00 UTC, create a PostgreSQL plain-SQL logical dump using the matching PostgreSQL 18 client, following Qlockify's pg_dump flow.
  • Backup package contains database archive plus non-secret manifest/checksums only.
  • Encrypt before durable local/remote storage using Qlockify's OpenSSL AES-256-CBC with PBKDF2 and 200,000 iterations; pair it with recorded SHA-256 checksums because CBC encryption is not itself authenticated.
  • Upload to private S3-compatible storage over TLS and verify the remote object.
  • Retain the latest three verified encrypted local backups and seven verified encrypted remote backups.
  • Delete stale backups only after a new remote backup is verified; never delete the last known-good copy.
  • Exclude .env, secrets, Redis, TLS keys, raw CSV, credentials, logs, monitoring state, font binary, and FontLicense.txt.
  • A dump is successful only after plain-SQL header/size/checksum validation, package-content inspection, encryption, upload, remote verification, and manifest completion.

Restore contract

  • Default restore target is a new empty non-production database; plain SQL is restored with psql -X --set ON_ERROR_STOP=on and a reviewed transaction strategy.
  • Validate explicit backup id, checksums, manifest, package contents, plain-SQL header, PostgreSQL compatibility, free space, and exact resolved destination before mutation.
  • Production restore additionally requires environment, host, database, backup id, typed confirmation, maintenance mode, exclusive lock, and named operator.
  • Never pipe an unchecked dump into the active database or restore over a running destination in place.
  • Restore into an empty database/cluster, validate counts/invariants/tenant isolation/login/cold Redis analytics/SSE, then switch/reopen traffic.
  • Rehearse monthly and before destructive migrations; record achieved RPO/RTO.

Safe first-server bootstrap

  1. Record provider/instance, OS, network, capacity, recovery console, operators, DNS, and rollback window.
  2. Prove two key-only operator logins and provider console recovery before disabling root/password SSH.
  3. Configure default-deny firewall: operator-allowlisted SSH plus public 80/443 only.
  4. Patch the OS, enable time sync/security updates/log rotation/disk monitoring, and reboot-test access.
  5. Verify the preinstalled Docker Engine/Compose installation, record its source/version, and ensure only the trusted ubuntu user has the required Docker-group access.
  6. Clone the deployment repository and backend/frontend nested repositories into exact validated paths.
  7. Create protected configuration/secret paths outside Git and validate permissions.
  8. Validate Compose and local builds without starting public traffic.
  9. Point DNS, test ACME staging, issue production TLS, and verify canonical routing.
  10. Run tracked scripts/first-start.sh: start data services, migrate, import/reconcile the pinned archive, then start API/worker/frontend/Nginx and run health checks.
  11. Run separate demo credential generation, authenticate as at least two merchants, and verify tenant isolation.
  12. Complete encrypted remote backup and clean restore rehearsal before handoff.

Threats and failure behavior

Failure/threat Required behavior
Wrong server/path/repository Stop before Git, build, volume, or database mutation
Dirty/diverged checkout Stop and ask operator; never unconditional hard reset
Pull/build failure Keep current containers/release; record evidence
Missing/malformed secret/config Fail before mutation without printing value
PostgreSQL/Redis unhealthy Do not start/route dependent candidate services
Migration failure Stop release; no candidate backend/worker; no automatic down migration
Import failure Mark incomplete; keep normal startup independent; no scheduling
Frontend/font failure Keep previous frontend; no deployment-side asset workaround
Worker/Redis loss Preserve PostgreSQL; reconcile/recompute idempotently
Certificate failure Production readiness fails; do not serve plaintext application traffic
Backup/upload failure Preserve all prior verified copies and alert/operator-block destructive work
Restore target mismatch Abort before database mutation
Disk/log exhaustion risk Bounded logs/resources and stop deploy before critical free-space floor
Spoofed operator Key-only restricted SSH, named release record, minimal Docker access
Secret disclosure File permissions, log allowlists, no env in backup; rotate first if exposed

Always / Ask / Never boundaries

Always

  • Verify exact server, paths, repository remotes/branches/cleanliness, and record all three commit hashes before build.
  • Build from the recorded checkouts and record the resulting release.
  • Keep Nginx as the only public application entry point and use valid HTTPS.
  • Separate migrations, API startup, worker startup, CSV import, and credential generation.
  • Keep Asynq pinned to pre-1.0 v0.26.0 until compatibility/recovery tests approve a change.
  • Bound resources, timeouts, retries, logs, and backup retention.
  • Back up PostgreSQL encrypted off-host and rehearse restore.
  • Use the ordinary frontend build context for the tracked font/license.

Ask first

  • Expose a new port, hostname, admin/debug/docs/metrics route, or third-party endpoint.
  • Change PostgreSQL/Redis major version, backup provider/encryption, retention, RPO/RTO, or restore behavior.
  • Run a destructive/non-compatible migration, production restore, or replacement import.
  • Change the manual latest-main deployment rule, repository topology, deploy-user privilege, resource limits, exact image tags, or TLS/HSTS scope.
  • Add Prometheus/Grafana before MVP is ready or start AI before the observability phase is accepted.
  • Change frontend ownership/location of the licensed assets.

Never

  • Add a CI/CD pipeline, runner, automated deployment trigger, or private image registry without a new approved decision.
  • Push from an agent session or rewrite/discard user history.
  • Publish PostgreSQL, Redis, backend, worker, monitoring, or Docker daemon ports to the internet.
  • Use unconditional git reset --hard on a dirty/diverged server checkout.
  • Run migration or CSV import inside API startup; schedule/watch/poll the CSV.
  • Include .env, secrets, TLS keys, Redis, raw CSV, credentials, logs, or font/license files in database backup.
  • Log credentials, raw tokens/cookies, payer-card keys, raw payment rows, or secret connection values.
  • Automatically run down migrations or restore over production without every confirmation guard.
  • Add another font delivery/injection path or copy the assets into deployment Git.
  • Disable health/security/backup gates to finish a deployment.

Measurable acceptance criteria

Manual deployment and rollback

  • All three nested repositories resolve to approved remotes, latest origin/main, clean working trees, and recorded commit hashes before build.
  • An induced pull/build failure leaves the active stack unchanged.
  • A release record maps the running stack to three Git hashes, local image ids, migration version, operator, backup, and health evidence without secrets.
  • Selecting the prior release hashes, rebuilding, and running checks completes application rollback within 15 minutes under normal conditions.
  • No CI/CD workflow, runner configuration, registry dependency, or automated deployment trigger exists.

Topology, routing, and lifecycle

  • External scanning finds only approved SSH plus TCP 80/443; all internal services are unreachable directly.
  • HTTP redirects to https://zarinpal.amiirkhl.ir; unknown hosts are rejected; TLS issuance and renewal pass.
  • /api/ reaches only backend, application routes reach frontend, and SSE events/heartbeats are delivered without buffering.
  • Clean startup health-gates PostgreSQL/Redis, runs migration once, then starts API/worker/frontend/Nginx.
  • Normal startup/deploy never runs import/admin. Identical explicit import does not duplicate business rows.
  • Redis deletion causes safe recomputation without loss of PostgreSQL payment/session data.
  • Required font URL returns 200 with correct type, while deployment Git contains no font/license copies.

Security and recovery

  • Real secrets remain outside Git with restrictive permissions and do not appear in Compose output/logs.
  • Docker logs are structured/bounded and an induced backend failure is traceable by request/release id without sensitive fields.
  • Daily backup contains only PostgreSQL archive plus non-secret manifest/checksums, is encrypted, remotely verified, and retained three local/seven remote.
  • No excluded .env, secret, Redis, TLS, CSV, credential, log, monitoring, font, or license asset appears in the backup.
  • Clean restore completes within two hours and passes schema/count/tenant/login/cold-cache/worker/SSE validation.
  • Every missing production restore confirmation factor aborts before mutation.

Phase ordering

  • MVP can be operated through health, logs, release records, disk/certificate/backup checks without Prometheus/Grafana.
  • Prometheus/Grafana/exporters/dashboards/alerts are implemented and accepted after MVP readiness and before any AI implementation begins.

Implementation readiness

No stakeholder-level Platform Operations decisions or real server values are required to plan or implement the deployment artifacts. Follow QLOCKIFY-ADAPTATION-CONTRACT.md for the exact source-file mapping and root/backend/frontend environment ownership.

Tracked examples use blank secret placeholders and safe public defaults. The real server IP/DNS state, ParsPack values, operator identity, and secret values are runtime inputs supplied only when the relevant first-server or production operation is performed. Backend/frontend-owned command and variable names are consumed from their implementation contracts during the corresponding deployment slice; they do not reopen the approved topology or block earlier independent slices.

Implementation remains gated by explicit stakeholder authorization. If a discovered interface would change an approved behavior rather than fill in a service-owned command/value, amend this spec first.

References

Reference in New Issue View Git Blame Copy Permalink