|
|
|
|
@@ -1,6 +1,6 @@
|
|
|
|
|
# Platform Operations Capability Spec
|
|
|
|
|
|
|
|
|
|
- Status: Draft revision — confirmed intent, ready for detailed stakeholder review
|
|
|
|
|
- Status: Approved specification
|
|
|
|
|
- Implementation authorization: Not granted
|
|
|
|
|
- Capability: Platform operations
|
|
|
|
|
- Depends on: all deployable backend and frontend capability contracts
|
|
|
|
|
@@ -57,19 +57,19 @@ This observability phase must be complete before the AI capability starts. It do
|
|
|
|
|
|
|
|
|
|
## Approved assumptions and decisions
|
|
|
|
|
|
|
|
|
|
1. The target is one supported Ubuntu server.
|
|
|
|
|
2. A dedicated trusted Ubuntu deployment user belongs to the `docker` group and runs Docker Compose. Docker-group membership is root-equivalent and is restricted to trusted operators.
|
|
|
|
|
1. The target is a new dedicated Ubuntu 24.04 LTS server with 2 CPU cores, 2 GB RAM, Docker already installed, user `ubuntu`, and a bootstrap-created 2 GB swap file.
|
|
|
|
|
2. The trusted `ubuntu` user belongs to the `docker` group and runs Docker Compose. Docker-group membership is root-equivalent, so access to this account is restricted to trusted operators.
|
|
|
|
|
3. The deployment repository contains ignored nested checkouts of the backend and frontend repositories, matching ADR 0001 and Qlockify.
|
|
|
|
|
4. Normal deployment selects the latest protected `main` from all three repositories, then records the exact three commit hashes before building.
|
|
|
|
|
4. The operator manually runs `git pull --ff-only origin main` in all three repositories and enters HTTPS credentials interactively. The deployment command performs no pull or credential handling; it verifies and records the three commits before building.
|
|
|
|
|
5. Application images are built on the production server from those recorded checkouts. They are not published to a registry.
|
|
|
|
|
6. Rollback checks out the previous recorded hashes and rebuilds. Database rollback remains constrained by migration compatibility.
|
|
|
|
|
7. Nginx obtains and renews a Let's Encrypt certificate automatically through Certbot using HTTP-01.
|
|
|
|
|
8. PostgreSQL 18 and Redis 7 Alpine are the initial major versions. Exact patch tags are pinned before implementation.
|
|
|
|
|
9. Redis is disposable cache/queue/coordination state and is excluded from backup. PostgreSQL remains authoritative.
|
|
|
|
|
8. Infrastructure images are pinned to `postgres:18.4-alpine3.24`, `redis:7.4.10-alpine3.21`, `nginx:1.29.8-alpine3.23`, and `certbot/certbot:v5.7.0`.
|
|
|
|
|
9. Redis uses a named volume, AOF `everysec`, and `noeviction` for routine restart continuity. It remains disposable cache/queue/coordination state and is excluded from backup; PostgreSQL remains authoritative.
|
|
|
|
|
10. Daily encrypted PostgreSQL backups and tested restores are MVP requirements.
|
|
|
|
|
11. Qlockify backup/restore/upload scripts are the implementation reference and will receive only necessary project path, service, credential, and safety changes.
|
|
|
|
|
12. Production `.env` files, secrets, TLS keys, Redis data, raw CSV, credentials, logs, and font assets are excluded from database backup packages.
|
|
|
|
|
13. Prometheus/Grafana are required after MVP readiness and before AI implementation.
|
|
|
|
|
13. Prometheus/Grafana are required after MVP readiness and before AI implementation. MVP readiness requires application flows, tenant isolation, migration/import, TLS, manual health/log checks, rollback, encrypted backup, and clean restore to pass.
|
|
|
|
|
14. Frontend commit `64f5ac2` owns `static/fonts/IRANYekanXVFaNumVF.woff2` and `static/fonts/FontLicense.txt`; deployment uses the normal frontend build context without another asset path.
|
|
|
|
|
|
|
|
|
|
Any change to these decisions requires a spec revision before implementation.
|
|
|
|
|
@@ -77,7 +77,7 @@ Any change to these decisions requires a spec revision before implementation.
|
|
|
|
|
## Repository and server layout
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
/home/<deploy-user>/zarinpal-analytics-deployment/
|
|
|
|
|
/home/ubuntu/zarinpal-analytics-deployment/
|
|
|
|
|
├── .git/ deployment repository
|
|
|
|
|
├── backend/ ignored checkout of backend repository
|
|
|
|
|
│ └── .git/
|
|
|
|
|
@@ -128,7 +128,7 @@ Nginx is the only public application entry point. The worker is event-driven and
|
|
|
|
|
| `backend` | Go API, sessions, authorization, analytics control | `app`, `data` | none | none |
|
|
|
|
|
| `worker` | On-demand Asynq analytics sections | `data` | none | none |
|
|
|
|
|
| `postgres` | Authoritative database | `data` | `postgres_data` | none |
|
|
|
|
|
| `redis` | Queue, cache, rate-limit, replay coordination | `data` | optional routine AOF volume, excluded from backup | none |
|
|
|
|
|
| `redis` | Queue, cache, rate-limit, replay coordination | `data` | named AOF volume, excluded from backup | none |
|
|
|
|
|
| `migrate` | Versioned schema migration | `data` | none | one-off only |
|
|
|
|
|
| `import` | One-time CSV bootstrap/recovery | `data` | exact read-only input bind | one-off profile only |
|
|
|
|
|
| `admin` | Demo credentials and approved maintenance | `data` | explicit output bind if requested | one-off profile only |
|
|
|
|
|
@@ -137,7 +137,7 @@ Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate priv
|
|
|
|
|
|
|
|
|
|
### Image and build policy
|
|
|
|
|
|
|
|
|
|
- PostgreSQL, Redis, Nginx, Certbot, and later monitoring images use explicit supported tags; no `latest` tag.
|
|
|
|
|
- PostgreSQL, Redis, Nginx, and Certbot use the exact approved tags above; no `latest` tag.
|
|
|
|
|
- Backend and worker build from the same recorded backend commit and deployment Dockerfile/context contract.
|
|
|
|
|
- Frontend builds from the recorded frontend commit and normal build context, including its tracked font/license inputs.
|
|
|
|
|
- The deployment script records resulting local image identifiers and build timestamps for diagnosis, but Git commit hashes are the reproducible source reference.
|
|
|
|
|
@@ -152,15 +152,28 @@ Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate priv
|
|
|
|
|
- CPU, memory, PID, database-pool, and log limits are explicit. Initial values are validated against the actual server and import/load tests.
|
|
|
|
|
- Docker logs use bounded rotation so a noisy service cannot fill the server disk.
|
|
|
|
|
|
|
|
|
|
Initial runtime limits for the 2-GB server are:
|
|
|
|
|
|
|
|
|
|
| Service | Memory limit |
|
|
|
|
|
|---|---:|
|
|
|
|
|
| PostgreSQL | 512 MB |
|
|
|
|
|
| Redis | 256 MB container limit; 192 MB `maxmemory` target to leave process/AOF overhead |
|
|
|
|
|
| Backend API | 256 MB |
|
|
|
|
|
| Worker | 512 MB |
|
|
|
|
|
| Frontend | 96 MB |
|
|
|
|
|
| Nginx | 96 MB |
|
|
|
|
|
|
|
|
|
|
Certbot is one-shot/periodic rather than continuously memory-resident. Worker concurrency remains 16 globally and 4 per merchant. If measured load exceeds this server, upgrade server capacity rather than silently lowering the approved concurrency.
|
|
|
|
|
|
|
|
|
|
## Manual deployment and release records
|
|
|
|
|
|
|
|
|
|
### Normal deployment
|
|
|
|
|
|
|
|
|
|
1. Confirm exact server, deployment root, environment, available disk, recent backup, Docker health, and operator identity.
|
|
|
|
|
2. Acquire a deployment lock so backup/restore/import/deploy operations cannot conflict unsafely.
|
|
|
|
|
3. Verify all three repositories exist at their exact expected paths, use approved remotes, and contain no unexplained local changes.
|
|
|
|
|
4. Fetch and fast-forward each repository to its latest `origin/main`. Do not discard local/user changes automatically.
|
|
|
|
|
5. Record the deployment, backend, and frontend commit hashes before building.
|
|
|
|
|
3. Before running the deployment command, the operator manually executes `git pull --ff-only origin main` in each repository and enters HTTPS credentials interactively; no Git credential is stored by deployment.
|
|
|
|
|
4. Verify all three repositories exist at their exact expected paths, use the approved HTTPS remotes, are on `main`, match `origin/main`, and contain no unexplained local changes.
|
|
|
|
|
5. Record the deployment, backend, and frontend commit hashes under the ignored `releases/` directory before building.
|
|
|
|
|
6. Validate required configuration/secret files without printing values.
|
|
|
|
|
7. Render and validate Docker Compose configuration.
|
|
|
|
|
8. Build candidate application/Nginx images on the server.
|
|
|
|
|
@@ -223,10 +236,12 @@ Compose health/dependency conditions prevent common startup races, but the manua
|
|
|
|
|
### One-time CSV import
|
|
|
|
|
|
|
|
|
|
- Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
|
|
|
|
|
- On the first server release only, tracked `scripts/first-start.sh` orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, and then public application startup. Later ordinary deploys/restarts never invoke it.
|
|
|
|
|
- The operator provides an exact resolved path outside Git; only that file is mounted read-only.
|
|
|
|
|
- Import cannot overlap migration, restore, or another import.
|
|
|
|
|
- The backend importer validates checksum/schema, streams data, reconciles counts, and records incomplete failure safely according to the Data Foundation spec.
|
|
|
|
|
- Import cannot overlap restore, backup, or another import and starts only after migration succeeds.
|
|
|
|
|
- The backend importer pins the approved checksum/schema/filename/counts, enforces resource caps, streams data, reconciles counts, removes temporary raw staging, and records incomplete failure safely according to the approved backend import specifications.
|
|
|
|
|
- An identical completed import is idempotent and does not duplicate business rows.
|
|
|
|
|
- Import failure prevents the first public application activation. Recovery restarts from the immutable archive; it never resumes guessed partial CSV/database state.
|
|
|
|
|
- Demo credential generation is a separate one-time `admin` job whose plaintext output never enters Compose logs, Git, backups, or shell history by default.
|
|
|
|
|
|
|
|
|
|
### Graceful shutdown
|
|
|
|
|
@@ -256,7 +271,7 @@ Initial health timing target is 10-second interval, 3-second timeout, 30-second
|
|
|
|
|
### Routing
|
|
|
|
|
|
|
|
|
|
- Port 80 serves `/.well-known/acme-challenge/` and redirects everything else to `https://zarinpal.amiirkhl.ir$request_uri`.
|
|
|
|
|
- Alternate approved hostnames redirect to the canonical hostname. Unknown hosts are rejected by a default server.
|
|
|
|
|
- The certificate and application serve only `zarinpal.amiirkhl.ir`; there is no `www` or alternate domain. Unknown hosts are rejected by a default server.
|
|
|
|
|
- `/api/` proxies only to the backend with host, scheme, safe client-address chain, and request id.
|
|
|
|
|
- `/` serves/proxies the frontend; SPA fallback never captures API errors.
|
|
|
|
|
- Fingerprinted assets receive long immutable caching; HTML/mutable manifests revalidate.
|
|
|
|
|
@@ -282,11 +297,11 @@ Initial health timing target is 10-second interval, 3-second timeout, 30-second
|
|
|
|
|
## Configuration and secrets
|
|
|
|
|
|
|
|
|
|
- A committed safe example names required variables and contains placeholders only.
|
|
|
|
|
- Real production `.env` and secret files live outside public Git with mode 0600 or stricter.
|
|
|
|
|
- Follow the Qlockify environment layout: deployment `.env` in the deployment root plus backend/frontend `.env` files in their nested repository roots. All are Git-ignored, mode 0600, and excluded from backups.
|
|
|
|
|
- Secrets are mounted only into the services that need them and never passed in command-line arguments or embedded in images.
|
|
|
|
|
- Frontend receives no production secret.
|
|
|
|
|
- Compose validation and diagnostics must not print resolved secret values.
|
|
|
|
|
- Backup object-store credentials and encryption passphrase are read from protected server configuration, following the Qlockify script interface with renamed ZarinPal variables/paths as needed.
|
|
|
|
|
- Backup object-store credentials and encryption passphrase are read from the protected deployment `.env`, following the Qlockify script interface with ZarinPal-specific ParsPack bucket/prefix and separate credentials.
|
|
|
|
|
- If a secret reaches Git/logs, rotate/revoke it first, then clean history/evidence as a separate incident action.
|
|
|
|
|
|
|
|
|
|
### Licensed font
|
|
|
|
|
@@ -362,19 +377,19 @@ Necessary changes include deployment root, archive/prefix names, Compose service
|
|
|
|
|
|
|
|
|
|
### Backup contract
|
|
|
|
|
|
|
|
|
|
- Daily PostgreSQL logical dump using a matching PostgreSQL 18 client.
|
|
|
|
|
- Daily at `02:00 UTC`, create a PostgreSQL plain-SQL logical dump using the matching PostgreSQL 18 client, following Qlockify's `pg_dump` flow.
|
|
|
|
|
- Backup package contains database archive plus non-secret manifest/checksums only.
|
|
|
|
|
- Encrypt before durable local/remote storage using the approved Qlockify-compatible mechanism.
|
|
|
|
|
- Encrypt before durable local/remote storage using Qlockify's OpenSSL AES-256-CBC with PBKDF2 and 200,000 iterations; pair it with recorded SHA-256 checksums because CBC encryption is not itself authenticated.
|
|
|
|
|
- Upload to private S3-compatible storage over TLS and verify the remote object.
|
|
|
|
|
- Retain the latest three verified encrypted local backups and seven verified encrypted remote backups.
|
|
|
|
|
- Delete stale backups only after a new remote backup is verified; never delete the last known-good copy.
|
|
|
|
|
- Exclude `.env`, secrets, Redis, TLS keys, raw CSV, credentials, logs, monitoring state, font binary, and `FontLicense.txt`.
|
|
|
|
|
- A dump is successful only after archive listing/checksum, encryption, upload, remote verification, and manifest completion.
|
|
|
|
|
- A dump is successful only after plain-SQL header/size/checksum validation, package-content inspection, encryption, upload, remote verification, and manifest completion.
|
|
|
|
|
|
|
|
|
|
### Restore contract
|
|
|
|
|
|
|
|
|
|
- Default restore target is a new empty non-production database.
|
|
|
|
|
- Validate explicit backup id, checksums, manifest, archive listing, PostgreSQL compatibility, free space, and exact resolved destination before mutation.
|
|
|
|
|
- Default restore target is a new empty non-production database; plain SQL is restored with `psql -X --set ON_ERROR_STOP=on` and a reviewed transaction strategy.
|
|
|
|
|
- Validate explicit backup id, checksums, manifest, package contents, plain-SQL header, PostgreSQL compatibility, free space, and exact resolved destination before mutation.
|
|
|
|
|
- Production restore additionally requires environment, host, database, backup id, typed confirmation, maintenance mode, exclusive lock, and named operator.
|
|
|
|
|
- Never pipe an unchecked dump into the active database or restore over a running destination in place.
|
|
|
|
|
- Restore into an empty database/cluster, validate counts/invariants/tenant isolation/login/cold Redis analytics/SSE, then switch/reopen traffic.
|
|
|
|
|
@@ -386,13 +401,13 @@ Necessary changes include deployment root, archive/prefix names, Compose service
|
|
|
|
|
2. Prove two key-only operator logins and provider console recovery before disabling root/password SSH.
|
|
|
|
|
3. Configure default-deny firewall: operator-allowlisted SSH plus public 80/443 only.
|
|
|
|
|
4. Patch the OS, enable time sync/security updates/log rotation/disk monitoring, and reboot-test access.
|
|
|
|
|
5. Install Docker Engine/Compose from the approved source and add only the dedicated trusted deploy user to the `docker` group.
|
|
|
|
|
5. Verify the preinstalled Docker Engine/Compose installation, record its source/version, and ensure only the trusted `ubuntu` user has the required Docker-group access.
|
|
|
|
|
6. Clone the deployment repository and backend/frontend nested repositories into exact validated paths.
|
|
|
|
|
7. Create protected configuration/secret paths outside Git and validate permissions.
|
|
|
|
|
8. Validate Compose and local builds without starting public traffic.
|
|
|
|
|
9. Point DNS, test ACME staging, issue production TLS, and verify canonical routing.
|
|
|
|
|
10. Start data services, run migration, then API/worker/frontend/Nginx and health checks.
|
|
|
|
|
11. Run the explicit one-time CSV import and separate credential generation only after the platform is healthy.
|
|
|
|
|
10. Run tracked `scripts/first-start.sh`: start data services, migrate, import/reconcile the pinned archive, then start API/worker/frontend/Nginx and run health checks.
|
|
|
|
|
11. Run separate demo credential generation, authenticate as at least two merchants, and verify tenant isolation.
|
|
|
|
|
12. Complete encrypted remote backup and clean restore rehearsal before handoff.
|
|
|
|
|
|
|
|
|
|
## Threats and failure behavior
|
|
|
|
|
@@ -433,7 +448,7 @@ Necessary changes include deployment root, archive/prefix names, Compose service
|
|
|
|
|
- Expose a new port, hostname, admin/debug/docs/metrics route, or third-party endpoint.
|
|
|
|
|
- Change PostgreSQL/Redis major version, backup provider/encryption, retention, RPO/RTO, or restore behavior.
|
|
|
|
|
- Run a destructive/non-compatible migration, production restore, or replacement import.
|
|
|
|
|
- Change the manual latest-`main` deployment rule, repository topology, deploy-user privilege, or TLS/HSTS scope.
|
|
|
|
|
- Change the manual latest-`main` deployment rule, repository topology, deploy-user privilege, resource limits, exact image tags, or TLS/HSTS scope.
|
|
|
|
|
- Add Prometheus/Grafana before MVP is ready or start AI before the observability phase is accepted.
|
|
|
|
|
- Change frontend ownership/location of the licensed assets.
|
|
|
|
|
|
|
|
|
|
@@ -484,17 +499,16 @@ Necessary changes include deployment root, archive/prefix names, Compose service
|
|
|
|
|
- MVP can be operated through health, logs, release records, disk/certificate/backup checks without Prometheus/Grafana.
|
|
|
|
|
- Prometheus/Grafana/exporters/dashboards/alerts are implemented and accepted after MVP readiness and before any AI implementation begins.
|
|
|
|
|
|
|
|
|
|
## Remaining decisions before implementation approval
|
|
|
|
|
## Remaining implementation inputs
|
|
|
|
|
|
|
|
|
|
1. Confirm server provider, Ubuntu version, CPU/RAM/disk, public IP/DNS owner, and deployment username/path.
|
|
|
|
|
2. Approve exact PostgreSQL 18, Redis 7 Alpine, Nginx, Certbot, and base-image tags after compatibility/security checks.
|
|
|
|
|
3. Approve initial CPU, memory, PID, database-pool, log-rotation, and disk thresholds for the selected server.
|
|
|
|
|
4. Confirm exact backend/frontend repository URLs, build contexts, commands, health endpoints, migration/import/admin commands, and SSE path.
|
|
|
|
|
5. Confirm S3-compatible endpoint/bucket/prefix, new ZarinPal credentials, encryption passphrase custody, and cron time.
|
|
|
|
|
6. Confirm whether Redis uses an AOF named volume for routine restart continuity or starts fully disposable; it remains excluded from backup either way.
|
|
|
|
|
7. Approve the exact deployment release-record location/format and the safe behavior for a dirty or divergent server checkout.
|
|
|
|
|
8. Confirm who may deploy/restore and where off-server copies of production configuration/secrets are escrowed.
|
|
|
|
|
9. Define the explicit “MVP ready” gate that starts the Prometheus/Grafana phase and the observability acceptance gate that permits AI work.
|
|
|
|
|
No stakeholder-level Platform Operations decisions remain. Implementation still consumes these concrete inputs without reopening the approved design:
|
|
|
|
|
|
|
|
|
|
- the new server's public IP and DNS access
|
|
|
|
|
- actual ZarinPal ParsPack endpoint/bucket/prefix credentials and separately protected encryption passphrase
|
|
|
|
|
- backend/frontend build commands, health endpoints, migration/import/admin commands, and exact SSE path from their approved implementation contracts
|
|
|
|
|
- the named operators allowed to perform deployment/restore and the off-server secret/configuration escrow location
|
|
|
|
|
|
|
|
|
|
If an input would change an approved behavior rather than fill in a value/interface, amend this spec first.
|
|
|
|
|
|
|
|
|
|
## References
|
|
|
|
|
|
|
|
|
|
|
