diff --git a/README.md b/README.md index 9c0081e..5f26500 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ zarinpal-analytics-deployment/ # this repository ├── backend/ # ignored checkout of zarinpal-analytics-backend ├── frontend/ # ignored checkout of zarinpal-analytics-frontend ├── docs/ -└── (deployment assets after specification approval) +└── (deployment assets only after explicit build authorization) ``` The intended production hostname is `zarinpal.amiirkhl.ir`. The deployment shape will follow the proven Qlockify structure while being reviewed and hardened for this service; it is not a byte-for-byte copy. @@ -26,7 +26,7 @@ The intended production hostname is `zarinpal.amiirkhl.ir`. The deployment shape ## Platform operations documents -- `docs/PLATFORM-OPERATIONS-SPEC.md` defines the draft manual Qlockify-style production topology, lifecycle, observability phases, security, backup/restore, acceptance, and operator boundaries. +- `docs/PLATFORM-OPERATIONS-SPEC.md` defines the approved manual Qlockify-style production topology, lifecycle, observability phases, security, backup/restore, acceptance, and operator boundaries. - `docs/runbooks/FIRST-SERVER-BOOTSTRAP.md` defines the safe first-host sequence without implementing a bootstrap script. - `docs/runbooks/BACKUP-RESTORE.md` defines logical PostgreSQL backup, encryption/retention, guarded restore, and rehearsal. - `tasks/plan.md` and `tasks/todo.md` hold the approval-gated implementation plan. Their presence does not grant implementation authorization. diff --git a/docs/PLATFORM-OPERATIONS-SPEC.md b/docs/PLATFORM-OPERATIONS-SPEC.md index 70e4bca..e64dcc6 100644 --- a/docs/PLATFORM-OPERATIONS-SPEC.md +++ b/docs/PLATFORM-OPERATIONS-SPEC.md @@ -1,6 +1,6 @@ # Platform Operations Capability Spec -- Status: Draft revision — confirmed intent, ready for detailed stakeholder review +- Status: Approved specification - Implementation authorization: Not granted - Capability: Platform operations - Depends on: all deployable backend and frontend capability contracts @@ -57,19 +57,19 @@ This observability phase must be complete before the AI capability starts. It do ## Approved assumptions and decisions -1. The target is one supported Ubuntu server. -2. A dedicated trusted Ubuntu deployment user belongs to the `docker` group and runs Docker Compose. Docker-group membership is root-equivalent and is restricted to trusted operators. +1. The target is a new dedicated Ubuntu 24.04 LTS server with 2 CPU cores, 2 GB RAM, Docker already installed, user `ubuntu`, and a bootstrap-created 2 GB swap file. +2. The trusted `ubuntu` user belongs to the `docker` group and runs Docker Compose. Docker-group membership is root-equivalent, so access to this account is restricted to trusted operators. 3. The deployment repository contains ignored nested checkouts of the backend and frontend repositories, matching ADR 0001 and Qlockify. -4. Normal deployment selects the latest protected `main` from all three repositories, then records the exact three commit hashes before building. +4. The operator manually runs `git pull --ff-only origin main` in all three repositories and enters HTTPS credentials interactively. The deployment command performs no pull or credential handling; it verifies and records the three commits before building. 5. Application images are built on the production server from those recorded checkouts. They are not published to a registry. 6. Rollback checks out the previous recorded hashes and rebuilds. Database rollback remains constrained by migration compatibility. 7. Nginx obtains and renews a Let's Encrypt certificate automatically through Certbot using HTTP-01. -8. PostgreSQL 18 and Redis 7 Alpine are the initial major versions. Exact patch tags are pinned before implementation. -9. Redis is disposable cache/queue/coordination state and is excluded from backup. PostgreSQL remains authoritative. +8. Infrastructure images are pinned to `postgres:18.4-alpine3.24`, `redis:7.4.10-alpine3.21`, `nginx:1.29.8-alpine3.23`, and `certbot/certbot:v5.7.0`. +9. Redis uses a named volume, AOF `everysec`, and `noeviction` for routine restart continuity. It remains disposable cache/queue/coordination state and is excluded from backup; PostgreSQL remains authoritative. 10. Daily encrypted PostgreSQL backups and tested restores are MVP requirements. 11. Qlockify backup/restore/upload scripts are the implementation reference and will receive only necessary project path, service, credential, and safety changes. 12. Production `.env` files, secrets, TLS keys, Redis data, raw CSV, credentials, logs, and font assets are excluded from database backup packages. -13. Prometheus/Grafana are required after MVP readiness and before AI implementation. +13. Prometheus/Grafana are required after MVP readiness and before AI implementation. MVP readiness requires application flows, tenant isolation, migration/import, TLS, manual health/log checks, rollback, encrypted backup, and clean restore to pass. 14. Frontend commit `64f5ac2` owns `static/fonts/IRANYekanXVFaNumVF.woff2` and `static/fonts/FontLicense.txt`; deployment uses the normal frontend build context without another asset path. Any change to these decisions requires a spec revision before implementation. @@ -77,7 +77,7 @@ Any change to these decisions requires a spec revision before implementation. ## Repository and server layout ```text -/home//zarinpal-analytics-deployment/ +/home/ubuntu/zarinpal-analytics-deployment/ ├── .git/ deployment repository ├── backend/ ignored checkout of backend repository │ └── .git/ @@ -128,7 +128,7 @@ Nginx is the only public application entry point. The worker is event-driven and | `backend` | Go API, sessions, authorization, analytics control | `app`, `data` | none | none | | `worker` | On-demand Asynq analytics sections | `data` | none | none | | `postgres` | Authoritative database | `data` | `postgres_data` | none | -| `redis` | Queue, cache, rate-limit, replay coordination | `data` | optional routine AOF volume, excluded from backup | none | +| `redis` | Queue, cache, rate-limit, replay coordination | `data` | named AOF volume, excluded from backup | none | | `migrate` | Versioned schema migration | `data` | none | one-off only | | `import` | One-time CSV bootstrap/recovery | `data` | exact read-only input bind | one-off profile only | | `admin` | Demo credentials and approved maintenance | `data` | explicit output bind if requested | one-off profile only | @@ -137,7 +137,7 @@ Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate priv ### Image and build policy -- PostgreSQL, Redis, Nginx, Certbot, and later monitoring images use explicit supported tags; no `latest` tag. +- PostgreSQL, Redis, Nginx, and Certbot use the exact approved tags above; no `latest` tag. - Backend and worker build from the same recorded backend commit and deployment Dockerfile/context contract. - Frontend builds from the recorded frontend commit and normal build context, including its tracked font/license inputs. - The deployment script records resulting local image identifiers and build timestamps for diagnosis, but Git commit hashes are the reproducible source reference. @@ -152,15 +152,28 @@ Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate priv - CPU, memory, PID, database-pool, and log limits are explicit. Initial values are validated against the actual server and import/load tests. - Docker logs use bounded rotation so a noisy service cannot fill the server disk. +Initial runtime limits for the 2-GB server are: + +| Service | Memory limit | +|---|---:| +| PostgreSQL | 512 MB | +| Redis | 256 MB container limit; 192 MB `maxmemory` target to leave process/AOF overhead | +| Backend API | 256 MB | +| Worker | 512 MB | +| Frontend | 96 MB | +| Nginx | 96 MB | + +Certbot is one-shot/periodic rather than continuously memory-resident. Worker concurrency remains 16 globally and 4 per merchant. If measured load exceeds this server, upgrade server capacity rather than silently lowering the approved concurrency. + ## Manual deployment and release records ### Normal deployment 1. Confirm exact server, deployment root, environment, available disk, recent backup, Docker health, and operator identity. 2. Acquire a deployment lock so backup/restore/import/deploy operations cannot conflict unsafely. -3. Verify all three repositories exist at their exact expected paths, use approved remotes, and contain no unexplained local changes. -4. Fetch and fast-forward each repository to its latest `origin/main`. Do not discard local/user changes automatically. -5. Record the deployment, backend, and frontend commit hashes before building. +3. Before running the deployment command, the operator manually executes `git pull --ff-only origin main` in each repository and enters HTTPS credentials interactively; no Git credential is stored by deployment. +4. Verify all three repositories exist at their exact expected paths, use the approved HTTPS remotes, are on `main`, match `origin/main`, and contain no unexplained local changes. +5. Record the deployment, backend, and frontend commit hashes under the ignored `releases/` directory before building. 6. Validate required configuration/secret files without printing values. 7. Render and validate Docker Compose configuration. 8. Build candidate application/Nginx images on the server. @@ -223,10 +236,12 @@ Compose health/dependency conditions prevent common startup races, but the manua ### One-time CSV import - Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger. +- On the first server release only, tracked `scripts/first-start.sh` orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, and then public application startup. Later ordinary deploys/restarts never invoke it. - The operator provides an exact resolved path outside Git; only that file is mounted read-only. -- Import cannot overlap migration, restore, or another import. -- The backend importer validates checksum/schema, streams data, reconciles counts, and records incomplete failure safely according to the Data Foundation spec. +- Import cannot overlap restore, backup, or another import and starts only after migration succeeds. +- The backend importer pins the approved checksum/schema/filename/counts, enforces resource caps, streams data, reconciles counts, removes temporary raw staging, and records incomplete failure safely according to the approved backend import specifications. - An identical completed import is idempotent and does not duplicate business rows. +- Import failure prevents the first public application activation. Recovery restarts from the immutable archive; it never resumes guessed partial CSV/database state. - Demo credential generation is a separate one-time `admin` job whose plaintext output never enters Compose logs, Git, backups, or shell history by default. ### Graceful shutdown @@ -256,7 +271,7 @@ Initial health timing target is 10-second interval, 3-second timeout, 30-second ### Routing - Port 80 serves `/.well-known/acme-challenge/` and redirects everything else to `https://zarinpal.amiirkhl.ir$request_uri`. -- Alternate approved hostnames redirect to the canonical hostname. Unknown hosts are rejected by a default server. +- The certificate and application serve only `zarinpal.amiirkhl.ir`; there is no `www` or alternate domain. Unknown hosts are rejected by a default server. - `/api/` proxies only to the backend with host, scheme, safe client-address chain, and request id. - `/` serves/proxies the frontend; SPA fallback never captures API errors. - Fingerprinted assets receive long immutable caching; HTML/mutable manifests revalidate. @@ -282,11 +297,11 @@ Initial health timing target is 10-second interval, 3-second timeout, 30-second ## Configuration and secrets - A committed safe example names required variables and contains placeholders only. -- Real production `.env` and secret files live outside public Git with mode 0600 or stricter. +- Follow the Qlockify environment layout: deployment `.env` in the deployment root plus backend/frontend `.env` files in their nested repository roots. All are Git-ignored, mode 0600, and excluded from backups. - Secrets are mounted only into the services that need them and never passed in command-line arguments or embedded in images. - Frontend receives no production secret. - Compose validation and diagnostics must not print resolved secret values. -- Backup object-store credentials and encryption passphrase are read from protected server configuration, following the Qlockify script interface with renamed ZarinPal variables/paths as needed. +- Backup object-store credentials and encryption passphrase are read from the protected deployment `.env`, following the Qlockify script interface with ZarinPal-specific ParsPack bucket/prefix and separate credentials. - If a secret reaches Git/logs, rotate/revoke it first, then clean history/evidence as a separate incident action. ### Licensed font @@ -362,19 +377,19 @@ Necessary changes include deployment root, archive/prefix names, Compose service ### Backup contract -- Daily PostgreSQL logical dump using a matching PostgreSQL 18 client. +- Daily at `02:00 UTC`, create a PostgreSQL plain-SQL logical dump using the matching PostgreSQL 18 client, following Qlockify's `pg_dump` flow. - Backup package contains database archive plus non-secret manifest/checksums only. -- Encrypt before durable local/remote storage using the approved Qlockify-compatible mechanism. +- Encrypt before durable local/remote storage using Qlockify's OpenSSL AES-256-CBC with PBKDF2 and 200,000 iterations; pair it with recorded SHA-256 checksums because CBC encryption is not itself authenticated. - Upload to private S3-compatible storage over TLS and verify the remote object. - Retain the latest three verified encrypted local backups and seven verified encrypted remote backups. - Delete stale backups only after a new remote backup is verified; never delete the last known-good copy. - Exclude `.env`, secrets, Redis, TLS keys, raw CSV, credentials, logs, monitoring state, font binary, and `FontLicense.txt`. -- A dump is successful only after archive listing/checksum, encryption, upload, remote verification, and manifest completion. +- A dump is successful only after plain-SQL header/size/checksum validation, package-content inspection, encryption, upload, remote verification, and manifest completion. ### Restore contract -- Default restore target is a new empty non-production database. -- Validate explicit backup id, checksums, manifest, archive listing, PostgreSQL compatibility, free space, and exact resolved destination before mutation. +- Default restore target is a new empty non-production database; plain SQL is restored with `psql -X --set ON_ERROR_STOP=on` and a reviewed transaction strategy. +- Validate explicit backup id, checksums, manifest, package contents, plain-SQL header, PostgreSQL compatibility, free space, and exact resolved destination before mutation. - Production restore additionally requires environment, host, database, backup id, typed confirmation, maintenance mode, exclusive lock, and named operator. - Never pipe an unchecked dump into the active database or restore over a running destination in place. - Restore into an empty database/cluster, validate counts/invariants/tenant isolation/login/cold Redis analytics/SSE, then switch/reopen traffic. @@ -386,13 +401,13 @@ Necessary changes include deployment root, archive/prefix names, Compose service 2. Prove two key-only operator logins and provider console recovery before disabling root/password SSH. 3. Configure default-deny firewall: operator-allowlisted SSH plus public 80/443 only. 4. Patch the OS, enable time sync/security updates/log rotation/disk monitoring, and reboot-test access. -5. Install Docker Engine/Compose from the approved source and add only the dedicated trusted deploy user to the `docker` group. +5. Verify the preinstalled Docker Engine/Compose installation, record its source/version, and ensure only the trusted `ubuntu` user has the required Docker-group access. 6. Clone the deployment repository and backend/frontend nested repositories into exact validated paths. 7. Create protected configuration/secret paths outside Git and validate permissions. 8. Validate Compose and local builds without starting public traffic. 9. Point DNS, test ACME staging, issue production TLS, and verify canonical routing. -10. Start data services, run migration, then API/worker/frontend/Nginx and health checks. -11. Run the explicit one-time CSV import and separate credential generation only after the platform is healthy. +10. Run tracked `scripts/first-start.sh`: start data services, migrate, import/reconcile the pinned archive, then start API/worker/frontend/Nginx and run health checks. +11. Run separate demo credential generation, authenticate as at least two merchants, and verify tenant isolation. 12. Complete encrypted remote backup and clean restore rehearsal before handoff. ## Threats and failure behavior @@ -433,7 +448,7 @@ Necessary changes include deployment root, archive/prefix names, Compose service - Expose a new port, hostname, admin/debug/docs/metrics route, or third-party endpoint. - Change PostgreSQL/Redis major version, backup provider/encryption, retention, RPO/RTO, or restore behavior. - Run a destructive/non-compatible migration, production restore, or replacement import. -- Change the manual latest-`main` deployment rule, repository topology, deploy-user privilege, or TLS/HSTS scope. +- Change the manual latest-`main` deployment rule, repository topology, deploy-user privilege, resource limits, exact image tags, or TLS/HSTS scope. - Add Prometheus/Grafana before MVP is ready or start AI before the observability phase is accepted. - Change frontend ownership/location of the licensed assets. @@ -484,17 +499,16 @@ Necessary changes include deployment root, archive/prefix names, Compose service - MVP can be operated through health, logs, release records, disk/certificate/backup checks without Prometheus/Grafana. - Prometheus/Grafana/exporters/dashboards/alerts are implemented and accepted after MVP readiness and before any AI implementation begins. -## Remaining decisions before implementation approval +## Remaining implementation inputs -1. Confirm server provider, Ubuntu version, CPU/RAM/disk, public IP/DNS owner, and deployment username/path. -2. Approve exact PostgreSQL 18, Redis 7 Alpine, Nginx, Certbot, and base-image tags after compatibility/security checks. -3. Approve initial CPU, memory, PID, database-pool, log-rotation, and disk thresholds for the selected server. -4. Confirm exact backend/frontend repository URLs, build contexts, commands, health endpoints, migration/import/admin commands, and SSE path. -5. Confirm S3-compatible endpoint/bucket/prefix, new ZarinPal credentials, encryption passphrase custody, and cron time. -6. Confirm whether Redis uses an AOF named volume for routine restart continuity or starts fully disposable; it remains excluded from backup either way. -7. Approve the exact deployment release-record location/format and the safe behavior for a dirty or divergent server checkout. -8. Confirm who may deploy/restore and where off-server copies of production configuration/secrets are escrowed. -9. Define the explicit “MVP ready” gate that starts the Prometheus/Grafana phase and the observability acceptance gate that permits AI work. +No stakeholder-level Platform Operations decisions remain. Implementation still consumes these concrete inputs without reopening the approved design: + +- the new server's public IP and DNS access +- actual ZarinPal ParsPack endpoint/bucket/prefix credentials and separately protected encryption passphrase +- backend/frontend build commands, health endpoints, migration/import/admin commands, and exact SSE path from their approved implementation contracts +- the named operators allowed to perform deployment/restore and the off-server secret/configuration escrow location + +If an input would change an approved behavior rather than fill in a value/interface, amend this spec first. ## References diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index e6565ec..f559f25 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -60,8 +60,12 @@ Framework decisions remain subordinate to capability requirements. SvelteKit is - Nginx TLS module: - PostgreSQL backup and restore: - PostgreSQL SQL dumps: +- PostgreSQL official image tags: +- Redis official image tags: +- Nginx official image tags: +- Certbot official image tags: -Applied in `PLATFORM-OPERATIONS-SPEC.md` through health-gated Compose startup, isolated networks and secrets, bounded container logs, TLS/SSE proxy behavior, and logical PostgreSQL backup with guarded restore. Exact product versions and server resource limits remain explicit approval decisions rather than assumptions. +Applied in `PLATFORM-OPERATIONS-SPEC.md` through health-gated Compose startup, isolated networks and secrets, bounded container logs, TLS/SSE proxy behavior, logical PostgreSQL backup with guarded restore, and the approved exact image tags and resource limits. ## Qlockify deployment reference diff --git a/docs/decisions/0002-manual-server-build-releases.md b/docs/decisions/0002-manual-server-build-releases.md index 9248431..6fb1008 100644 --- a/docs/decisions/0002-manual-server-build-releases.md +++ b/docs/decisions/0002-manual-server-build-releases.md @@ -15,18 +15,19 @@ The platform still needs to answer which source is running and how to return to Use manual Qlockify-style deployment on the production server. -For a normal release, a trusted deployment user: +For a normal release, the trusted `ubuntu` user: 1. verifies the exact deployment/backend/frontend repository paths, remotes, branches, and clean working trees; -2. fetches and fast-forwards each checkout to the latest `origin/main`; -3. records the exact three resulting Git commit hashes; -4. builds application images locally with Docker Compose; -5. runs the dedicated migration and health/smoke checks; and -6. records the release as active only after validation passes. +2. manually runs `git pull --ff-only origin main` in each checkout, entering the source-forge username/password interactively when prompted; +3. invokes the deployment command, which verifies that all three checkouts are clean, on `main`, and equal to `origin/main` without performing any Git pull; +4. records the exact three resulting Git commit hashes under the ignored `/home/ubuntu/zarinpal-analytics-deployment/releases/` directory; +5. builds application images locally with Docker Compose; +6. runs the dedicated migration and health/smoke checks; and +7. records the release as active only after validation passes. Rollback selects the previous release record, checks out its three hashes, rebuilds locally, and repeats the checks. Database down-migration is never automatic; migrations follow expand/contract compatibility. -The deploy flow may adapt Qlockify's `deploy.sh`, but it must not unconditionally `git reset --hard`. A dirty or divergent checkout stops for operator resolution so user changes are not destroyed. +The deploy flow may adapt Qlockify's `deploy.sh`, but Git pulling is a separate operator prerequisite and the deployment command must not store source-forge credentials. It must not unconditionally `git reset --hard`. A dirty or divergent checkout stops for operator resolution so user changes are not destroyed. There is no CI/CD workflow, Actions runner, automated cross-repository event, private image registry, or direct automated production mutation.