starcoach-hackathon/zarinpal-analytics-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document setup and deployment

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-23 01:40:45 +08:00
parent 5911bfb275
commit 72238e7e87
1 changed files with 287 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

319
README.md
View File

@@ -1,54 +1,309 @@
# ZarinPal Merchant Analytics — Backend
Go backend for merchant identity, dataset ingestion, deterministic analytics, authenticated APIs, and the bounded AvalAI-assisted growth analyst.
Go backend for merchant authentication, one-time payment-data ingestion, deterministic analytics, Redis-coordinated background jobs, and optional AI-assisted insights.
Current state: the local MVP implements database migration, streaming dataset import,
secure demo identity, deterministic dashboard analytics, merchant-scoped details/CSV,
the authenticated HTTP API, and a Redis-connected worker boundary.
## What runs in this repository
## Local development
| Command | Purpose | Run mode |
| ------------------------------ | ----------------------------------------------------------------------- | ------------------------------------------------- |
| `go run ./cmd/migrate` | Apply versioned PostgreSQL migrations | One-off, before the first start and after updates |
| `go run ./cmd/importer` | Import the immutable `challenge_data.csv.gz` dataset | One-off bootstrap/recovery command |
| `go run ./cmd/admin` | Create or reset the demo login from `DEMO_USERNAME` and `DEMO_PASSWORD` | One-off, after importing data |
| `go run ./cmd/api` | Start the HTTP API on `HTTP_ADDR` | Long-running |
| `go run ./cmd/worker` | Process queued growth reports, page insights, and growth-coach jobs | Long-running alongside the API |
| `go run ./cmd/scheduler` | Enqueue due growth-coach work when `AI_COACH_ENABLED=true` | Optional long-running process |
| `go run ./cmd/calendar --help` | Import, approve, or activate reviewed calendar catalogs | Administrative/optional |
| `go run ./cmd/admin-reset-ai` | Reset AI-related administrative state | Administrative/recovery only |
Copy `.env.example` to the ignored `.env`, load its variables in your shell, and run:
The importer is checksum-idempotent: importing the same archive again is a safe no-op. It is not a file watcher or scheduled ETL process. The worker does not replace the API; both processes should run at the same time.
```text
go run ./cmd/migrate
go run ./cmd/importer
go run ./cmd/admin
go run ./cmd/worker
go run ./cmd/api
## Local setup
### Prerequisites
- Go 1.24 or newer
- PostgreSQL 18 (a compatible recent PostgreSQL release may also work)
- Redis 7.4
- Git
- The source dataset as a gzip-compressed CSV
PostgreSQL stores application and analytics data. Redis is disposable coordination/cache state used by the API and worker. Both services must be reachable before starting the backend.
### 1. Start PostgreSQL and Redis
Use locally installed services, or start development-only containers:
```bash
docker run --name zpa-postgres \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=change-me \
-e POSTGRES_DB=zarinpal \
-p 127.0.0.1:5432:5432 \
-d postgres:18.4-alpine3.24
docker run --name zpa-redis \
-p 127.0.0.1:6379:6379 \
-d redis:7.4.10-alpine3.21 redis-server --appendonly yes
```
`cmd/admin` idempotently creates or resets `DEMO_USERNAME` using `DEMO_PASSWORD`
and assigns it to the imported merchant with the largest session population. It
never prints the password or stores it outside the Argon2id hash. The local API
uses the `zpa_session` HttpOnly cookie when `COOKIE_SECURE=false`; secure
deployments use the contract's `__Host-zpa_session` cookie.
On later runs, restart them with:
## Local dataset input
```bash
docker start zpa-postgres zpa-redis
```
Place the provided archive at:
These ports bind to localhost only. Do not expose PostgreSQL or Redis directly to the internet.
### 2. Configure the backend
From the repository root:
```bash
cp .env.example .env
```
PowerShell equivalent:
```powershell
Copy-Item .env.example .env
```
Edit `.env` and replace all placeholder values. For the documented local demo login, use:
```dotenv
DATABASE_URL=postgresql://postgres:change-me@localhost:5432/zarinpal?sslmode=disable
REDIS_ADDR=localhost:6379
REDIS_PASSWORD=
DEMO_USERNAME=merchant-0001
DEMO_PASSWORD=merchant-demo-2026
SESSION_SECRET=<at-least-32-random-characters>
CURSOR_SECRET=<a-different-at-least-32-character-secret>
FRONTEND_ORIGIN=http://127.0.0.1:5173
COOKIE_SECURE=false
```
Generate independent secrets, for example with `openssl rand -hex 32`. The Go commands read process environment variables; they do not automatically load `.env`.
Load `.env` in Bash before running commands:
```bash
set -a
source .env
set +a
```
For PowerShell, load the file into the current terminal:
```powershell
Get-Content .env | ForEach-Object {
if ($_ -match '^\s*([^#][^=]*)=(.*)$') {
[Environment]::SetEnvironmentVariable($matches[1].Trim(), $matches[2], 'Process')
}
}
```
Keep `.env` private and uncommitted. AI features are optional and disabled in the example. If enabled, the provider key belongs only in the backend environment—never in the frontend.
### 3. Add the dataset
Place the archive at the configured `IMPORT_PATH`, which defaults to:
```text
data/incoming/challenge_data.csv.gz
```
The entire `data/incoming/` payload is ignored by Git. The one-time bootstrap importer will read from this location (or an explicitly supplied path), stream the compressed CSV, validate it, and write an import manifest containing the source checksum, schema version, row counts, quality results, and importer revision. It is an administrative bootstrap/recovery command, not a scheduled ingestion job. The hackathon MVP does not mutate payment facts after this import; any future ZarinPal live-write integration is outside this project's current scope. The raw file and resulting PostgreSQL data are not committed.
The dataset directory is Git-ignored. Do not commit the archive, database dumps, import output, or credentials.
The proposed first-server Docker flow and Bash entrypoint are specified in [`docs/IMPORT-WORKFLOW-SPEC.md`](docs/IMPORT-WORKFLOW-SPEC.md). Implementation is authorized and must follow the dependency order in [`tasks/todo.md`](tasks/todo.md).
### 4. Prepare the database and demo user
## Planned runnable boundaries
Run these commands from the repository root and in this order:
One repository may expose several small `cmd/` entry points while sharing domain packages under `internal/`:
```bash
go mod download
go run ./cmd/migrate
go run ./cmd/importer
go run ./cmd/admin
```
- HTTP API
- dataset importer
- database migration command
- HTTP API with authenticated SSE analytics delivery
- on-demand analytics worker that calculates dashboard sections in parallel and caches versioned results in Redis
`cmd/admin` idempotently creates or resets the configured demo user and attaches it to the imported merchant with the largest session population. Passwords are stored as Argon2id hashes and are not printed by the command.
The approved package-layout and manual-constructor decisions are implemented through the scaffold and provider tasks.
Local demo credentials:
## Merchant API contract
```text
Username: merchant-0001
Password: merchant-demo-2026
```
- [`docs/API-SPEC.md`](docs/API-SPEC.md) defines the reviewable MVP capability, security boundaries, SSE recovery, detail pagination, and CSV-export policy.
- [`docs/openapi.yaml`](docs/openapi.yaml) is the approved OpenAPI 3.1 interface source of truth for generated frontend types.
These credentials work only when `.env` contains the same values before `cmd/admin` runs. They are public development credentials; change them for every internet-accessible deployment.
### 5. Start the worker and API
Open two terminals, load `.env` in both, and run:
```bash
# Terminal 1: background jobs
go run ./cmd/worker
# Terminal 2: HTTP API
go run ./cmd/api
```
The API is available at `http://127.0.0.1:8080`. Start the frontend separately at `http://127.0.0.1:5173`; its Vite server proxies `/api` to this API.
If the growth-coach scheduler is enabled, run it in a third terminal:
```bash
go run ./cmd/scheduler
```
## Verification and common commands
```bash
curl http://127.0.0.1:8080/healthz
go test ./...
go vet ./...
go build ./cmd/...
```
The equivalent Make targets are `make migrate`, `make import`, `make seed`, `make worker`, `make api`, `make test`, `make vet`, and `make build`.
## Production deployment on a remote server
Deploy the backend and frontend together through the `zarinpal-analytics-deployment` repository. Its Compose stack runs PostgreSQL, Redis, the API, worker, one-off tools, and the static frontend. A separate shared Caddy instance owns ports 80/443, TLS, and domain routing.
The examples below use `analytics.example.com`; replace it with your domain. Replace repository URL placeholders with your own template/fork URLs.
### 1. Prepare the server and DNS access
1. Provision an Ubuntu server with a public IP and enough disk/RAM for the dataset and PostgreSQL.
2. Install Git, Docker Engine, and the Docker Compose plugin.
3. Allow SSH only from trusted addresses and allow inbound TCP 80 and 443.
4. Do not publish PostgreSQL port 5432 or Redis port 6379.
### 2. Install the shared Caddy router
Use the [Caddy deployment repository](https://git.amiirkhl.ir/Interanet/caddy-deployment.git) to install the main router:
```bash
git clone https://git.amiirkhl.ir/Interanet/caddy-deployment.git ~/caddy-deployment
cd ~/caddy-deployment
```
Follow that repository's README to start Caddy. Ensure its Compose configuration creates/uses an external Docker network named `caddy_proxy` and attaches the Caddy container to it. If the repository does not create the network, create it once with `docker network create caddy_proxy` before starting Caddy.
### 3. Clone the three application repositories
The deployment script expects the backend and frontend as independent nested checkouts:
```bash
git clone <deployment-repository-url> ~/zarinpal-analytics-deployment
cd ~/zarinpal-analytics-deployment
git clone <backend-repository-url> backend
git clone <frontend-repository-url> frontend
```
All three repositories must be on a clean `main` branch matching `origin/main`.
### 4. Create production environment files
```bash
cp .env.example .env
cp backend/.env.production.example backend/.env
cp frontend/.env.production.example frontend/.env
chmod 600 .env backend/.env frontend/.env
```
Then:
- Replace every `change-me` value.
- Set `FRONTEND_ORIGIN=https://analytics.example.com` in `backend/.env`.
- Keep `COOKIE_SECURE=true`.
- Generate different PostgreSQL, Redis, session, and cursor secrets with `openssl rand -hex 32`.
- Set a new production `DEMO_PASSWORD`; do not reuse `merchant-demo-2026` on a public server.
- Leave AI flags off unless a backend-only provider key, model, and positive budgets are configured.
- Put no secret in `frontend/.env`; all `PUBLIC_*` values are embedded in the browser build.
### 5. Place the import archive
Copy the immutable archive to a private host path and make `IMPORT_FILE` in the deployment `.env` point to it:
```bash
sudo install -d -m 750 -o "$USER" -g "$USER" /srv/zarinpal-data
cp /path/to/challenge_data.csv.gz /srv/zarinpal-data/challenge_data.csv.gz
chmod 440 /srv/zarinpal-data/challenge_data.csv.gz
```
### 6. Perform the first application start
From `~/zarinpal-analytics-deployment`:
```bash
docker compose config --quiet
docker compose build backend frontend
docker compose up -d postgres redis
docker compose --profile tools run --rm migrate
docker compose --profile import run --rm importer
docker compose --profile tools run --rm admin
docker compose up -d backend worker frontend
docker compose ps
```
Wait until `postgres`, `redis`, `backend`, `worker`, and `frontend` are healthy. Inspect failures with:
```bash
docker compose logs --tail=200 postgres redis backend worker frontend
```
### 7. Connect the domain through Caddy
Create an `A` record for `analytics.example.com` pointing to the server's public IPv4 address. Add an `AAAA` record only when IPv6 is configured and allowed through the firewall.
Add a site to the Caddy deployment's `Caddyfile` (or its equivalent templated configuration):
```caddyfile
analytics.example.com {
@backend path /api/* /healthz
handle @backend {
reverse_proxy zarinpal-backend:8080 {
flush_interval -1
}
}
handle {
reverse_proxy zarinpal-frontend:8080
}
}
```
Validate and reload Caddy using the commands documented by the Caddy repository. Caddy will obtain and renew HTTPS certificates automatically after public DNS resolves and ports 80/443 reach the server.
Verify:
```bash
curl -I https://analytics.example.com/
curl https://analytics.example.com/healthz
```
Then sign in with the production values of `DEMO_USERNAME` and `DEMO_PASSWORD` and verify that dashboard requests complete and the worker logs show no repeated failures.
### 8. Deploy later updates
```bash
cd ~/zarinpal-analytics-deployment
git pull --ff-only origin main
git -C backend pull --ff-only origin main
git -C frontend pull --ff-only origin main
bash scripts/deploy.sh
```
Normal updates apply migrations and restart the application, but do not rerun the importer or reset the demo password. Run those one-off profiles explicitly only when intended.
Never use `docker compose down -v` in production: `-v` deletes the PostgreSQL and Redis volumes. Back up PostgreSQL and rehearse restoration before treating the deployment as production-ready.
## API and architecture documentation
- [`docs/openapi.yaml`](docs/openapi.yaml) is the OpenAPI 3.1 contract used to generate frontend types.
- [`docs/API-SPEC.md`](docs/API-SPEC.md) documents authentication, tenant isolation, SSE, pagination, and exports.
- [`docs/IMPORT-WORKFLOW-SPEC.md`](docs/IMPORT-WORKFLOW-SPEC.md) documents the import boundary.
- [`docs/METRIC-CATALOGUE.md`](docs/METRIC-CATALOGUE.md) defines analytics metrics.