Compare commits
4 Commits
feature/ai
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 93e3ece390 | |||
| 36ac6ee34e | |||
| 6634b6c4af | |||
| 994eec33dc |
@@ -13,6 +13,12 @@ REDIS_PASSWORD=change-me-with-a-different-random-hex-value
|
||||
# Absolute host path to the immutable first-import archive.
|
||||
IMPORT_FILE=/srv/zarinpal-data/challenge_data.csv.gz
|
||||
|
||||
# Reviewed calendar bootstrap input and non-secret audit identities.
|
||||
CALENDAR_FILE=/srv/zarinpal-data/reviewed-calendar.json
|
||||
CALENDAR_CURRENT_JALALI_YEAR=1405
|
||||
CALENDAR_REVIEWER=change-me-independent-reviewer
|
||||
CALENDAR_OPERATOR=change-me-platform-operator
|
||||
|
||||
# Optional backup settings. These are reserved for the backup/restore scripts.
|
||||
S3_BACKUP_BUCKET=
|
||||
S3_BACKUP_PREFIX=backup/zarinpal-analytics
|
||||
|
||||
14
README.md
14
README.md
@@ -16,7 +16,7 @@ zarinpal-analytics-deployment/ # this repository
|
||||
└── (deployment assets created through the authorized task list)
|
||||
```
|
||||
|
||||
The production hostname is `zarinpal.amiirkhl.ir`. Caddy routes `/api/*` and `/healthz` to the Go API and every other path to the static frontend. PostgreSQL, Redis, the worker, and all one-off commands remain on the private Compose data network.
|
||||
The production hostname is `zarinpal.amiirkhl.ir`. Caddy routes `/api/*` and `/healthz` to the Go API and every other path to the static frontend. PostgreSQL, Redis, the worker, and all one-off commands remain on the private Compose data network; the worker additionally joins a dedicated bridge network for outbound AI-provider access without joining the shared Caddy proxy network.
|
||||
|
||||
## Required server layout
|
||||
|
||||
@@ -38,11 +38,11 @@ cp frontend/.env.production.example frontend/.env
|
||||
chmod 600 .env backend/.env frontend/.env
|
||||
```
|
||||
|
||||
Replace every `change-me` value. Generate URL-safe secrets with `openssl rand -hex 32`. The root file owns the PostgreSQL and Redis passwords; Compose injects their connection settings into the Go containers. The frontend file contains public build-time switches only.
|
||||
Replace every `change-me` value. Generate URL-safe secrets with `openssl rand -hex 32`. The root file owns the PostgreSQL and Redis passwords plus the reviewed-calendar path and non-secret audit identities; Compose injects their connection settings into the Go containers. The frontend file contains public build-time switches only.
|
||||
|
||||
## First production start
|
||||
|
||||
Before adding/reloading the Caddy route, validate and build the stack, import the immutable dataset, create the demo account, and only then start the public upstreams:
|
||||
Before adding/reloading the Caddy route, place the independently reviewed calendar JSON at `CALENDAR_FILE`, make it read-only, and confirm that its `sourceChecksum` is not a review placeholder. Then validate and build the stack, import the immutable dataset, bootstrap the reviewed calendar, create the demo account, and only then start the public upstreams:
|
||||
|
||||
```bash
|
||||
docker compose config --quiet
|
||||
@@ -50,6 +50,7 @@ docker compose build backend frontend
|
||||
docker compose up -d postgres redis
|
||||
docker compose --profile tools run --rm migrate
|
||||
docker compose --profile import run --rm importer
|
||||
docker compose --profile tools run --rm calendar
|
||||
docker compose --profile tools run --rm admin
|
||||
docker compose up -d backend worker frontend
|
||||
docker compose ps
|
||||
@@ -68,18 +69,19 @@ git -C frontend pull --ff-only origin main
|
||||
bash scripts/deploy.sh
|
||||
```
|
||||
|
||||
Normal deploys run migrations but never run the importer or admin command.
|
||||
Normal deploys run migrations but never run the importer, calendar bootstrap, or admin command.
|
||||
|
||||
## Re-running one-off tools
|
||||
|
||||
`IMPORT_FILE` in the root `.env` is the absolute host path of the immutable `challenge_data.csv.gz`. If recovery requires the idempotent importer or the demo account must be reset, run them explicitly:
|
||||
`IMPORT_FILE` in the root `.env` is the absolute host path of the immutable `challenge_data.csv.gz`; `CALENDAR_FILE` is the absolute path of the reviewed JSON catalog. If recovery requires an idempotent data/catalog bootstrap or the demo account must be reset, run the applicable command explicitly:
|
||||
|
||||
```bash
|
||||
docker compose --profile import run --rm importer
|
||||
docker compose --profile tools run --rm calendar
|
||||
docker compose --profile tools run --rm admin
|
||||
```
|
||||
|
||||
Both commands are explicit one-offs. Re-running the importer relies on the backend's checksum-based idempotency guard.
|
||||
All three commands are explicit one-offs. Calendar bootstrap resumes the same reviewed version safely and rejects placeholder checksums, conflicting metadata, retired versions, or replacement of a different active catalog. Future catalog rotation must use the separately reviewed import, approve, and activate workflow.
|
||||
|
||||
## Operations
|
||||
|
||||
|
||||
41
compose.yaml
41
compose.yaml
@@ -56,10 +56,13 @@ services:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
umask 077
|
||||
printf 'appendonly yes\nappendfsync everysec\nmaxmemory 192mb\nmaxmemory-policy noeviction\nrequirepass %s\n' "$$REDIS_PASSWORD" > /tmp/redis.conf
|
||||
chown redis:redis /data /tmp/redis.conf
|
||||
exec su-exec redis redis-server /tmp/redis.conf
|
||||
chown redis:redis /data
|
||||
exec /usr/bin/setpriv --reuid redis --regid redis --clear-groups \
|
||||
/bin/sh -ec '
|
||||
umask 077
|
||||
printf "appendonly yes\nappendfsync everysec\nmaxmemory 192mb\nmaxmemory-policy noeviction\nrequirepass %s\n" "$$REDIS_PASSWORD" > /data/redis.conf
|
||||
exec redis-server /data/redis.conf
|
||||
'
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
networks:
|
||||
@@ -113,6 +116,10 @@ services:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- data
|
||||
- egress
|
||||
dns:
|
||||
- 1.1.1.1
|
||||
- 8.8.8.8
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "kill -0 1"]
|
||||
interval: 10s
|
||||
@@ -185,9 +192,35 @@ services:
|
||||
- data
|
||||
profiles: ["tools"]
|
||||
|
||||
calendar:
|
||||
<<: *backend-runtime
|
||||
entrypoint: ["/app/calendar"]
|
||||
command:
|
||||
- -action
|
||||
- bootstrap
|
||||
- -file
|
||||
- /calendar/reviewed.json
|
||||
- -current-jalali-year
|
||||
- "${CALENDAR_CURRENT_JALALI_YEAR:-0}"
|
||||
- -reviewer
|
||||
- "${CALENDAR_REVIEWER:-}"
|
||||
- -actor
|
||||
- "${CALENDAR_OPERATOR:-}"
|
||||
restart: "no"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- ${CALENDAR_FILE:-/srv/zarinpal-data/reviewed-calendar.json}:/calendar/reviewed.json:ro
|
||||
networks:
|
||||
- data
|
||||
profiles: ["tools"]
|
||||
|
||||
networks:
|
||||
data:
|
||||
internal: true
|
||||
egress:
|
||||
driver: bridge
|
||||
caddy_proxy:
|
||||
external: true
|
||||
name: ${CADDY_NETWORK:-caddy_proxy}
|
||||
|
||||
@@ -11,7 +11,8 @@ RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/api ./c
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/worker ./cmd/worker \
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/migrate ./cmd/migrate \
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/importer ./cmd/importer \
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/admin ./cmd/admin
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/admin ./cmd/admin \
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/calendar ./cmd/calendar
|
||||
|
||||
FROM alpine:3.22
|
||||
|
||||
|
||||
@@ -133,6 +133,7 @@ Nginx is the only public application entry point. The worker is event-driven and
|
||||
| `redis` | Queue, cache, rate-limit, replay coordination | `data` | named AOF volume, excluded from backup | none |
|
||||
| `migrate` | Versioned schema migration | `data` | none | one-off only |
|
||||
| `import` | One-time CSV bootstrap/recovery | `data` | exact read-only input bind | one-off profile only |
|
||||
| `calendar` | Reviewed catalog bootstrap with audited state transitions | `data` | reviewed read-only JSON bind | one-off profile only |
|
||||
| `admin` | Demo credentials and approved maintenance | `data` | explicit output bind if requested | one-off profile only |
|
||||
|
||||
Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate private monitoring network. Grafana binds only to loopback/private operator access unless a later ADR approves a protected public route.
|
||||
@@ -238,7 +239,7 @@ Compose health/dependency conditions prevent common startup races, but the manua
|
||||
### One-time CSV import
|
||||
|
||||
- Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
|
||||
- On the first server release only, tracked `scripts/first-start.sh` orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, and then public application startup. Later ordinary deploys/restarts never invoke it.
|
||||
- On the first server release only, tracked `scripts/first-start.sh` orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, reviewed calendar bootstrap, and then public application startup. Later ordinary deploys/restarts never invoke it.
|
||||
- The operator provides an exact resolved path outside Git; only that file is mounted read-only.
|
||||
- Import cannot overlap restore, backup, or another import and starts only after migration succeeds.
|
||||
- The backend importer pins the approved checksum/schema/filename/counts, enforces resource caps, streams data, reconciles counts, removes temporary raw staging, and records incomplete failure safely according to the approved backend import specifications.
|
||||
|
||||
@@ -11,11 +11,11 @@ The production host already runs a dedicated Caddy Compose project that owns por
|
||||
|
||||
ZarinPal Analytics connects its frontend and API containers to the existing external `caddy_proxy` Docker network using the stable aliases `zarinpal-frontend` and `zarinpal-backend`. Caddy routes `zarinpal.amiirkhl.ir` to those aliases. This Compose project publishes no host ports and owns no TLS material.
|
||||
|
||||
PostgreSQL, Redis, the worker, migration, importer, and admin services remain on an internal data network. Only the API additionally joins `caddy_proxy`; the frontend joins only `caddy_proxy`. Caddy preserves the complete `/api` path and disables response buffering for proxied API traffic so authenticated SSE streams are delivered promptly.
|
||||
PostgreSQL, Redis, the worker, migration, importer, and admin services remain on an internal data network. The worker additionally joins a dedicated bridge egress network so approved external AI-provider calls and DNS resolution can succeed without attaching it to the shared proxy network. Only the API additionally joins `caddy_proxy`; the frontend joins only `caddy_proxy`. Caddy preserves the complete `/api` path and disables response buffering for proxied API traffic so authenticated SSE streams are delivered promptly.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The Caddy deployment must exist first and create the external `caddy_proxy` network.
|
||||
- Caddy configuration and this deployment must be released together for first activation.
|
||||
- Nginx/Certbot-specific requirements in the earlier platform documents are superseded by this ADR. Their routing, TLS, and port-ownership intent is retained and implemented by the shared Caddy service.
|
||||
- The shared proxy network is a trusted infrastructure boundary; no database, Redis, worker, or administrative service is attached to it.
|
||||
- The shared proxy network is a trusted infrastructure boundary; no database, Redis, worker, or administrative service is attached to it. Worker internet access is isolated through the dedicated `egress` network.
|
||||
|
||||
@@ -95,15 +95,16 @@ If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/c
|
||||
|
||||
1. Acquire the production deployment lock and record the operator/release id.
|
||||
2. Place the approved `challenge_data.csv.gz` at the exact Git-ignored deployment `data/incoming/` path. Verify regular-file/read-only properties, size, and pinned SHA-256; do not copy it into an image.
|
||||
3. Invoke tracked `scripts/first-start.sh`. It starts PostgreSQL and Redis, waits with bounded backoff, applies the one-off migration, runs the one-shot importer with a read-only mount, and reconciles the pinned counts before starting public services.
|
||||
4. Review the JSON/Markdown import report and durable import-run/issues metadata. Fatal validation, resource, migration, or reconciliation failure must exit nonzero before backend/frontend/Nginx activation.
|
||||
5. Verify backend and worker liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
|
||||
6. Verify the frontend static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
|
||||
7. Verify Nginx public generic health, canonical redirect, security headers, login page, SSE behavior, and that only 80/443 are public.
|
||||
8. Run demo credential generation as a separate explicit administrative job. Store one-time plaintext output outside Git/logs/backups.
|
||||
9. Authenticate as at least two merchant accounts, verify different dashboards and cross-tenant isolation, then remove or escrow plaintext credential output.
|
||||
10. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
|
||||
11. Record the active three-hash release only after all postflight checks pass.
|
||||
3. Place the independently reviewed calendar JSON at the absolute `CALENDAR_FILE` path. Verify it is a regular read-only file, record its checksum and reviewer, and reject candidate data whose `sourceChecksum` still contains a review placeholder.
|
||||
4. Invoke tracked `scripts/first-start.sh`. It starts PostgreSQL and Redis, waits with bounded backoff, applies the one-off migration, runs the one-shot importer and reconciliation, bootstraps the reviewed calendar through its audited `DRAFT`/`APPROVED`/`ACTIVE` states, and only then starts public services.
|
||||
5. Review the JSON/Markdown import report, durable import-run/issues metadata, active calendar version, and calendar audit records. Fatal validation, resource, migration, import, reconciliation, calendar, or scheduling failure must exit nonzero before backend/frontend/Nginx activation.
|
||||
6. Verify backend and worker liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
|
||||
7. Verify the frontend static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
|
||||
8. Verify Nginx public generic health, canonical redirect, security headers, login page, SSE behavior, and that only 80/443 are public.
|
||||
9. Run demo credential generation as a separate explicit administrative job. Store one-time plaintext output outside Git/logs/backups.
|
||||
10. Authenticate as at least two merchant accounts, verify different dashboards and cross-tenant isolation, then remove or escrow plaintext credential output.
|
||||
11. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
|
||||
12. Record the active three-hash release only after all postflight checks pass.
|
||||
|
||||
Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user