Compare commits
4 Commits
64404bb7cf
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 93e3ece390 | |||
| 36ac6ee34e | |||
| 6634b6c4af | |||
| 994eec33dc |
@@ -13,6 +13,12 @@ REDIS_PASSWORD=change-me-with-a-different-random-hex-value
|
|||||||
# Absolute host path to the immutable first-import archive.
|
# Absolute host path to the immutable first-import archive.
|
||||||
IMPORT_FILE=/srv/zarinpal-data/challenge_data.csv.gz
|
IMPORT_FILE=/srv/zarinpal-data/challenge_data.csv.gz
|
||||||
|
|
||||||
|
# Reviewed calendar bootstrap input and non-secret audit identities.
|
||||||
|
CALENDAR_FILE=/srv/zarinpal-data/reviewed-calendar.json
|
||||||
|
CALENDAR_CURRENT_JALALI_YEAR=1405
|
||||||
|
CALENDAR_REVIEWER=change-me-independent-reviewer
|
||||||
|
CALENDAR_OPERATOR=change-me-platform-operator
|
||||||
|
|
||||||
# Optional backup settings. These are reserved for the backup/restore scripts.
|
# Optional backup settings. These are reserved for the backup/restore scripts.
|
||||||
S3_BACKUP_BUCKET=
|
S3_BACKUP_BUCKET=
|
||||||
S3_BACKUP_PREFIX=backup/zarinpal-analytics
|
S3_BACKUP_PREFIX=backup/zarinpal-analytics
|
||||||
|
|||||||
14
README.md
14
README.md
@@ -16,7 +16,7 @@ zarinpal-analytics-deployment/ # this repository
|
|||||||
└── (deployment assets created through the authorized task list)
|
└── (deployment assets created through the authorized task list)
|
||||||
```
|
```
|
||||||
|
|
||||||
The production hostname is `zarinpal.amiirkhl.ir`. Caddy routes `/api/*` and `/healthz` to the Go API and every other path to the static frontend. PostgreSQL, Redis, the worker, and all one-off commands remain on the private Compose data network.
|
The production hostname is `zarinpal.amiirkhl.ir`. Caddy routes `/api/*` and `/healthz` to the Go API and every other path to the static frontend. PostgreSQL, Redis, the worker, and all one-off commands remain on the private Compose data network; the worker additionally joins a dedicated bridge network for outbound AI-provider access without joining the shared Caddy proxy network.
|
||||||
|
|
||||||
## Required server layout
|
## Required server layout
|
||||||
|
|
||||||
@@ -38,11 +38,11 @@ cp frontend/.env.production.example frontend/.env
|
|||||||
chmod 600 .env backend/.env frontend/.env
|
chmod 600 .env backend/.env frontend/.env
|
||||||
```
|
```
|
||||||
|
|
||||||
Replace every `change-me` value. Generate URL-safe secrets with `openssl rand -hex 32`. The root file owns the PostgreSQL and Redis passwords; Compose injects their connection settings into the Go containers. The frontend file contains public build-time switches only.
|
Replace every `change-me` value. Generate URL-safe secrets with `openssl rand -hex 32`. The root file owns the PostgreSQL and Redis passwords plus the reviewed-calendar path and non-secret audit identities; Compose injects their connection settings into the Go containers. The frontend file contains public build-time switches only.
|
||||||
|
|
||||||
## First production start
|
## First production start
|
||||||
|
|
||||||
Before adding/reloading the Caddy route, validate and build the stack, import the immutable dataset, create the demo account, and only then start the public upstreams:
|
Before adding/reloading the Caddy route, place the independently reviewed calendar JSON at `CALENDAR_FILE`, make it read-only, and confirm that its `sourceChecksum` is not a review placeholder. Then validate and build the stack, import the immutable dataset, bootstrap the reviewed calendar, create the demo account, and only then start the public upstreams:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose config --quiet
|
docker compose config --quiet
|
||||||
@@ -50,6 +50,7 @@ docker compose build backend frontend
|
|||||||
docker compose up -d postgres redis
|
docker compose up -d postgres redis
|
||||||
docker compose --profile tools run --rm migrate
|
docker compose --profile tools run --rm migrate
|
||||||
docker compose --profile import run --rm importer
|
docker compose --profile import run --rm importer
|
||||||
|
docker compose --profile tools run --rm calendar
|
||||||
docker compose --profile tools run --rm admin
|
docker compose --profile tools run --rm admin
|
||||||
docker compose up -d backend worker frontend
|
docker compose up -d backend worker frontend
|
||||||
docker compose ps
|
docker compose ps
|
||||||
@@ -68,18 +69,19 @@ git -C frontend pull --ff-only origin main
|
|||||||
bash scripts/deploy.sh
|
bash scripts/deploy.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Normal deploys run migrations but never run the importer or admin command.
|
Normal deploys run migrations but never run the importer, calendar bootstrap, or admin command.
|
||||||
|
|
||||||
## Re-running one-off tools
|
## Re-running one-off tools
|
||||||
|
|
||||||
`IMPORT_FILE` in the root `.env` is the absolute host path of the immutable `challenge_data.csv.gz`. If recovery requires the idempotent importer or the demo account must be reset, run them explicitly:
|
`IMPORT_FILE` in the root `.env` is the absolute host path of the immutable `challenge_data.csv.gz`; `CALENDAR_FILE` is the absolute path of the reviewed JSON catalog. If recovery requires an idempotent data/catalog bootstrap or the demo account must be reset, run the applicable command explicitly:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose --profile import run --rm importer
|
docker compose --profile import run --rm importer
|
||||||
|
docker compose --profile tools run --rm calendar
|
||||||
docker compose --profile tools run --rm admin
|
docker compose --profile tools run --rm admin
|
||||||
```
|
```
|
||||||
|
|
||||||
Both commands are explicit one-offs. Re-running the importer relies on the backend's checksum-based idempotency guard.
|
All three commands are explicit one-offs. Calendar bootstrap resumes the same reviewed version safely and rejects placeholder checksums, conflicting metadata, retired versions, or replacement of a different active catalog. Future catalog rotation must use the separately reviewed import, approve, and activate workflow.
|
||||||
|
|
||||||
## Operations
|
## Operations
|
||||||
|
|
||||||
|
|||||||
41
compose.yaml
41
compose.yaml
@@ -56,10 +56,13 @@ services:
|
|||||||
- /bin/sh
|
- /bin/sh
|
||||||
- -ec
|
- -ec
|
||||||
- |
|
- |
|
||||||
umask 077
|
chown redis:redis /data
|
||||||
printf 'appendonly yes\nappendfsync everysec\nmaxmemory 192mb\nmaxmemory-policy noeviction\nrequirepass %s\n' "$$REDIS_PASSWORD" > /tmp/redis.conf
|
exec /usr/bin/setpriv --reuid redis --regid redis --clear-groups \
|
||||||
chown redis:redis /data /tmp/redis.conf
|
/bin/sh -ec '
|
||||||
exec su-exec redis redis-server /tmp/redis.conf
|
umask 077
|
||||||
|
printf "appendonly yes\nappendfsync everysec\nmaxmemory 192mb\nmaxmemory-policy noeviction\nrequirepass %s\n" "$$REDIS_PASSWORD" > /data/redis.conf
|
||||||
|
exec redis-server /data/redis.conf
|
||||||
|
'
|
||||||
volumes:
|
volumes:
|
||||||
- redis_data:/data
|
- redis_data:/data
|
||||||
networks:
|
networks:
|
||||||
@@ -113,6 +116,10 @@ services:
|
|||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
networks:
|
networks:
|
||||||
- data
|
- data
|
||||||
|
- egress
|
||||||
|
dns:
|
||||||
|
- 1.1.1.1
|
||||||
|
- 8.8.8.8
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "kill -0 1"]
|
test: ["CMD-SHELL", "kill -0 1"]
|
||||||
interval: 10s
|
interval: 10s
|
||||||
@@ -185,9 +192,35 @@ services:
|
|||||||
- data
|
- data
|
||||||
profiles: ["tools"]
|
profiles: ["tools"]
|
||||||
|
|
||||||
|
calendar:
|
||||||
|
<<: *backend-runtime
|
||||||
|
entrypoint: ["/app/calendar"]
|
||||||
|
command:
|
||||||
|
- -action
|
||||||
|
- bootstrap
|
||||||
|
- -file
|
||||||
|
- /calendar/reviewed.json
|
||||||
|
- -current-jalali-year
|
||||||
|
- "${CALENDAR_CURRENT_JALALI_YEAR:-0}"
|
||||||
|
- -reviewer
|
||||||
|
- "${CALENDAR_REVIEWER:-}"
|
||||||
|
- -actor
|
||||||
|
- "${CALENDAR_OPERATOR:-}"
|
||||||
|
restart: "no"
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
volumes:
|
||||||
|
- ${CALENDAR_FILE:-/srv/zarinpal-data/reviewed-calendar.json}:/calendar/reviewed.json:ro
|
||||||
|
networks:
|
||||||
|
- data
|
||||||
|
profiles: ["tools"]
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
data:
|
data:
|
||||||
internal: true
|
internal: true
|
||||||
|
egress:
|
||||||
|
driver: bridge
|
||||||
caddy_proxy:
|
caddy_proxy:
|
||||||
external: true
|
external: true
|
||||||
name: ${CADDY_NETWORK:-caddy_proxy}
|
name: ${CADDY_NETWORK:-caddy_proxy}
|
||||||
|
|||||||
@@ -11,7 +11,8 @@ RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/api ./c
|
|||||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/worker ./cmd/worker \
|
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/worker ./cmd/worker \
|
||||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/migrate ./cmd/migrate \
|
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/migrate ./cmd/migrate \
|
||||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/importer ./cmd/importer \
|
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/importer ./cmd/importer \
|
||||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/admin ./cmd/admin
|
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/admin ./cmd/admin \
|
||||||
|
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/calendar ./cmd/calendar
|
||||||
|
|
||||||
FROM alpine:3.22
|
FROM alpine:3.22
|
||||||
|
|
||||||
|
|||||||
@@ -133,6 +133,7 @@ Nginx is the only public application entry point. The worker is event-driven and
|
|||||||
| `redis` | Queue, cache, rate-limit, replay coordination | `data` | named AOF volume, excluded from backup | none |
|
| `redis` | Queue, cache, rate-limit, replay coordination | `data` | named AOF volume, excluded from backup | none |
|
||||||
| `migrate` | Versioned schema migration | `data` | none | one-off only |
|
| `migrate` | Versioned schema migration | `data` | none | one-off only |
|
||||||
| `import` | One-time CSV bootstrap/recovery | `data` | exact read-only input bind | one-off profile only |
|
| `import` | One-time CSV bootstrap/recovery | `data` | exact read-only input bind | one-off profile only |
|
||||||
|
| `calendar` | Reviewed catalog bootstrap with audited state transitions | `data` | reviewed read-only JSON bind | one-off profile only |
|
||||||
| `admin` | Demo credentials and approved maintenance | `data` | explicit output bind if requested | one-off profile only |
|
| `admin` | Demo credentials and approved maintenance | `data` | explicit output bind if requested | one-off profile only |
|
||||||
|
|
||||||
Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate private monitoring network. Grafana binds only to loopback/private operator access unless a later ADR approves a protected public route.
|
Post-MVP adds `prometheus`, `grafana`, and approved exporters on a separate private monitoring network. Grafana binds only to loopback/private operator access unless a later ADR approves a protected public route.
|
||||||
@@ -238,7 +239,7 @@ Compose health/dependency conditions prevent common startup races, but the manua
|
|||||||
### One-time CSV import
|
### One-time CSV import
|
||||||
|
|
||||||
- Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
|
- Import is an explicit operator command/profile, never normal startup, cron, a scheduler, watcher, or deploy trigger.
|
||||||
- On the first server release only, tracked `scripts/first-start.sh` orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, and then public application startup. Later ordinary deploys/restarts never invoke it.
|
- On the first server release only, tracked `scripts/first-start.sh` orchestrates PostgreSQL/Redis readiness, the one-off migration, the importer profile, reconciliation, reviewed calendar bootstrap, and then public application startup. Later ordinary deploys/restarts never invoke it.
|
||||||
- The operator provides an exact resolved path outside Git; only that file is mounted read-only.
|
- The operator provides an exact resolved path outside Git; only that file is mounted read-only.
|
||||||
- Import cannot overlap restore, backup, or another import and starts only after migration succeeds.
|
- Import cannot overlap restore, backup, or another import and starts only after migration succeeds.
|
||||||
- The backend importer pins the approved checksum/schema/filename/counts, enforces resource caps, streams data, reconciles counts, removes temporary raw staging, and records incomplete failure safely according to the approved backend import specifications.
|
- The backend importer pins the approved checksum/schema/filename/counts, enforces resource caps, streams data, reconciles counts, removes temporary raw staging, and records incomplete failure safely according to the approved backend import specifications.
|
||||||
|
|||||||
@@ -11,11 +11,11 @@ The production host already runs a dedicated Caddy Compose project that owns por
|
|||||||
|
|
||||||
ZarinPal Analytics connects its frontend and API containers to the existing external `caddy_proxy` Docker network using the stable aliases `zarinpal-frontend` and `zarinpal-backend`. Caddy routes `zarinpal.amiirkhl.ir` to those aliases. This Compose project publishes no host ports and owns no TLS material.
|
ZarinPal Analytics connects its frontend and API containers to the existing external `caddy_proxy` Docker network using the stable aliases `zarinpal-frontend` and `zarinpal-backend`. Caddy routes `zarinpal.amiirkhl.ir` to those aliases. This Compose project publishes no host ports and owns no TLS material.
|
||||||
|
|
||||||
PostgreSQL, Redis, the worker, migration, importer, and admin services remain on an internal data network. Only the API additionally joins `caddy_proxy`; the frontend joins only `caddy_proxy`. Caddy preserves the complete `/api` path and disables response buffering for proxied API traffic so authenticated SSE streams are delivered promptly.
|
PostgreSQL, Redis, the worker, migration, importer, and admin services remain on an internal data network. The worker additionally joins a dedicated bridge egress network so approved external AI-provider calls and DNS resolution can succeed without attaching it to the shared proxy network. Only the API additionally joins `caddy_proxy`; the frontend joins only `caddy_proxy`. Caddy preserves the complete `/api` path and disables response buffering for proxied API traffic so authenticated SSE streams are delivered promptly.
|
||||||
|
|
||||||
## Consequences
|
## Consequences
|
||||||
|
|
||||||
- The Caddy deployment must exist first and create the external `caddy_proxy` network.
|
- The Caddy deployment must exist first and create the external `caddy_proxy` network.
|
||||||
- Caddy configuration and this deployment must be released together for first activation.
|
- Caddy configuration and this deployment must be released together for first activation.
|
||||||
- Nginx/Certbot-specific requirements in the earlier platform documents are superseded by this ADR. Their routing, TLS, and port-ownership intent is retained and implemented by the shared Caddy service.
|
- Nginx/Certbot-specific requirements in the earlier platform documents are superseded by this ADR. Their routing, TLS, and port-ownership intent is retained and implemented by the shared Caddy service.
|
||||||
- The shared proxy network is a trusted infrastructure boundary; no database, Redis, worker, or administrative service is attached to it.
|
- The shared proxy network is a trusted infrastructure boundary; no database, Redis, worker, or administrative service is attached to it. Worker internet access is isolated through the dedicated `egress` network.
|
||||||
|
|||||||
@@ -95,15 +95,16 @@ If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/c
|
|||||||
|
|
||||||
1. Acquire the production deployment lock and record the operator/release id.
|
1. Acquire the production deployment lock and record the operator/release id.
|
||||||
2. Place the approved `challenge_data.csv.gz` at the exact Git-ignored deployment `data/incoming/` path. Verify regular-file/read-only properties, size, and pinned SHA-256; do not copy it into an image.
|
2. Place the approved `challenge_data.csv.gz` at the exact Git-ignored deployment `data/incoming/` path. Verify regular-file/read-only properties, size, and pinned SHA-256; do not copy it into an image.
|
||||||
3. Invoke tracked `scripts/first-start.sh`. It starts PostgreSQL and Redis, waits with bounded backoff, applies the one-off migration, runs the one-shot importer with a read-only mount, and reconciles the pinned counts before starting public services.
|
3. Place the independently reviewed calendar JSON at the absolute `CALENDAR_FILE` path. Verify it is a regular read-only file, record its checksum and reviewer, and reject candidate data whose `sourceChecksum` still contains a review placeholder.
|
||||||
4. Review the JSON/Markdown import report and durable import-run/issues metadata. Fatal validation, resource, migration, or reconciliation failure must exit nonzero before backend/frontend/Nginx activation.
|
4. Invoke tracked `scripts/first-start.sh`. It starts PostgreSQL and Redis, waits with bounded backoff, applies the one-off migration, runs the one-shot importer and reconciliation, bootstraps the reviewed calendar through its audited `DRAFT`/`APPROVED`/`ACTIVE` states, and only then starts public services.
|
||||||
5. Verify backend and worker liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
|
5. Review the JSON/Markdown import report, durable import-run/issues metadata, active calendar version, and calendar audit records. Fatal validation, resource, migration, import, reconciliation, calendar, or scheduling failure must exit nonzero before backend/frontend/Nginx activation.
|
||||||
6. Verify the frontend static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
|
6. Verify backend and worker liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
|
||||||
7. Verify Nginx public generic health, canonical redirect, security headers, login page, SSE behavior, and that only 80/443 are public.
|
7. Verify the frontend static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
|
||||||
8. Run demo credential generation as a separate explicit administrative job. Store one-time plaintext output outside Git/logs/backups.
|
8. Verify Nginx public generic health, canonical redirect, security headers, login page, SSE behavior, and that only 80/443 are public.
|
||||||
9. Authenticate as at least two merchant accounts, verify different dashboards and cross-tenant isolation, then remove or escrow plaintext credential output.
|
9. Run demo credential generation as a separate explicit administrative job. Store one-time plaintext output outside Git/logs/backups.
|
||||||
10. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
|
10. Authenticate as at least two merchant accounts, verify different dashboards and cross-tenant isolation, then remove or escrow plaintext credential output.
|
||||||
11. Record the active three-hash release only after all postflight checks pass.
|
11. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
|
||||||
|
12. Record the active three-hash release only after all postflight checks pass.
|
||||||
|
|
||||||
Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration.
|
Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user