Compare commits

..

45 Commits

Author SHA1 Message Date
f5e2a21bd9 fix: simplify Persian week labels 2026-08-20 22:53:50 +03:30
6523e90a03 fix: expose complete dataset period catalog 2026-08-20 22:28:07 +03:30
d6de48fd15 fix: harden terminal-scoped analytics contracts
Keep JSON and bounded CSV exports on one canonical predicate, replace replayable offsets with encrypted tenant/filter-bound keyset cursors, and suppress merchant-grain peers under terminal scope. Validate period filters and align OpenAPI and capability specs with runtime behavior.
2026-08-20 21:29:18 +03:30
857f29635e feat: add terminal-scoped transaction analytics 2026-08-20 20:56:15 +03:30
f710762b90 feat: expose complete Iranian reporting periods 2026-08-20 20:47:14 +03:30
630341f7be docs: plan terminal-aware dashboard API 2026-08-20 20:44:23 +03:30
d1c7b34388 feat: deliver authenticated merchant analytics API 2026-08-20 19:57:32 +03:30
b30e3439a4 feat: seed secure demo merchant identity 2026-08-20 19:52:01 +03:30
cc7a4e4718 feat: stream and reconcile challenge dataset 2026-08-20 19:52:01 +03:30
961980d18c feat: establish Go service and database foundation 2026-08-20 19:33:38 +03:30
552cae5f9d docs(sdd): authorize backend implementation 2026-08-20 19:17:49 +03:30
b82ba0dc23 docs(plan): expand backend implementation roadmap 2026-08-20 19:00:07 +03:30
ec625252fb docs(cache): align immutable dataset namespace 2026-08-20 19:00:06 +03:30
a0e2b70cf4 docs(metrics): align catalogue with physical schema 2026-08-20 18:49:55 +03:30
f9356da179 docs(import): approve validation and recovery policy 2026-08-20 18:49:12 +03:30
7416a0859c docs(import): define validation and recovery policy 2026-08-20 18:46:56 +03:30
d978730202 docs(import): approve docker bootstrap workflow 2026-08-20 18:44:31 +03:30
8ff093b238 docs(import): specify one-time docker bootstrap 2026-08-20 18:40:31 +03:30
27903ff67e docs(cache): remove static dataset invalidation 2026-08-20 18:40:28 +03:30
f471eaf3a2 docs(api): approve implementation task breakdown 2026-08-20 18:33:24 +03:30
47a23f505e docs(api): approve implementation plan 2026-08-20 18:31:57 +03:30
1f06d97b3b docs(api): fix plan formatting 2026-08-20 18:30:33 +03:30
509f53cd4e docs(api): break down API implementation tasks 2026-08-20 18:30:14 +03:30
4b04f9c58a docs(api): plan merchant API implementation 2026-08-20 18:30:13 +03:30
520b39305d docs(api): approve merchant API contract 2026-08-20 18:27:07 +03:30
e8028d5f46 docs(api): approve request size limits 2026-08-20 18:25:15 +03:30
e54c8e8319 docs(api): set problem documentation namespace 2026-08-20 18:24:26 +03:30
ee93460182 docs(schema): define normalized postgres model 2026-08-20 18:23:37 +03:30
08ac99a5a6 docs(data): locate verification policy on terminals 2026-08-20 18:23:36 +03:30
af69f549c0 docs(api): expose scoped terminal references 2026-08-20 18:20:13 +03:30
20c9ba35a6 docs(api): set analytics audit retention 2026-08-20 18:18:45 +03:30
d133019a14 docs(api): localize CSV columns in Persian 2026-08-20 18:17:19 +03:30
8358cc4d14 docs(api): approve CSV export limits 2026-08-20 18:14:16 +03:30
b031cfe280 docs: link merchant API contract 2026-08-20 18:08:53 +03:30
dfe88a148a docs(api): add OpenAPI v1 interface contract 2026-08-20 18:08:37 +03:30
076ccf2d0f docs(api): define merchant analytics capability 2026-08-20 18:07:19 +03:30
03c26cc21a docs(analytics): approve actionable insight thresholds 2026-08-20 17:40:07 +03:30
96e045c4e1 docs: approve analytics presentation defaults 2026-08-20 17:29:27 +03:30
a70fc15517 docs: adopt CTO dataset semantics 2026-08-20 17:29:26 +03:30
5ef650dcf6 docs: approve identity specification 2026-08-20 17:29:25 +03:30
c8e04605ee docs: draft analytics engine capability 2026-08-20 17:21:26 +03:30
50030a20e8 docs: finalize demo session persistence 2026-08-20 17:21:25 +03:30
384ea54282 docs: keep demo identity replaceable 2026-08-20 17:14:40 +03:30
406cd9c0cf docs: define merchant session baseline 2026-08-20 17:06:43 +03:30
d579c2525d docs: approve data foundation specification 2026-08-20 17:06:42 +03:30
54 changed files with 7354 additions and 117 deletions

13
.env.example Normal file
View File

@@ -0,0 +1,13 @@
# Local development only. Copy to .env and replace secrets; .env is ignored.
APP_ENV=development
HTTP_ADDR=127.0.0.1:8080
DATABASE_URL=postgresql://postgres:change-me@localhost:5432/zarinpal?sslmode=disable
REDIS_ADDR=localhost:6379
REDIS_PASSWORD=
IMPORT_PATH=data/incoming/challenge_data.csv.gz
DEMO_USERNAME=merchant-0001
DEMO_PASSWORD=replace-with-at-least-12-characters
SESSION_SECRET=replace-with-at-least-32-random-characters
CURSOR_SECRET=replace-with-a-different-32-character-secret
FRONTEND_ORIGIN=http://127.0.0.1:5173
COOKIE_SECURE=false

2
.gitignore vendored
View File

@@ -13,7 +13,7 @@
/coverage/
*.test
*.out
*.exe
/vendor/
.idea/
.vscode/

View File

@@ -2,9 +2,9 @@
## Phase gate
This repository is in specification mode. Do not create `go.mod`, application packages, migrations, containers, or executable code until the relevant capability spec is approved.
This repository is authorized for SDD Phase 4 implementation as of 2026-08-20. Follow `tasks/todo.md` in dependency order and keep the approved capability specs authoritative. New capabilities or changes outside those contracts still require specification and approval before implementation.
For every Go-related task, always invoke and follow the `golang-how-to` skill, then the secondary Go skills it routes to. Before creating the Go project layout, ask the user to approve the architecture style and dependency-injection approach in that order.
For every Go-related task, always invoke and follow the `golang-how-to` skill, then the secondary Go skills it routes to. The capability-oriented modular-monolith architecture and manual-constructor dependency injection are approved; changing either requires a new ADR and user approval.
## Product invariants
@@ -12,13 +12,14 @@ For every Go-related task, always invoke and follow the `golang-how-to` skill, t
- The backend is the authority for authentication, authorization, tenant isolation, metric definitions, and data provenance.
- Every merchant-scoped query derives merchant identity from the authenticated server-side session; never trust a client-supplied merchant identifier as authorization.
- Cross-merchant results are anonymized same-category aggregates only. Exclude the active merchant and suppress the benchmark unless at least 10 other merchants and 1,000 combined peer sessions contribute in the selected period. Use one value per peer merchant to calculate the cohort median and direction-aware performance percentile; never transaction-weight by default.
- Treat `payer_card_key` as an opaque, merchant-scoped source token with unverified upstream semantics. Retain it unchanged only for internal deterministic analytics; never expose it through merchant APIs, logs, exports, or AI inputs, and never describe it as a card number or verified person identifier.
- Treat `payer_card_key` as a pseudonymous card identifier whose equality is valid only inside one merchant. Retain it unchanged only for internal deterministic analytics; never expose it through merchant APIs, logs, exports, or AI inputs, and never describe it as a card number, cross-merchant identity, or verified person.
- Use integer minor currency units or an explicitly documented fixed-precision representation; never floating point for money.
- Interpret timezone-naive challenge timestamps as `Asia/Tehran`, preserve their source provenance, and normalize canonical database/API timestamps to UTC `timestamptz`/ISO 8601. Any change to this assumption requires a new import version and ADR.
- Raw imports are immutable inputs; corrections are new import versions, not silent mutation.
- CSV ingestion is a one-time bootstrap/recovery command only. Never add a file watcher, polling schedule, recurring CSV ETL, or ownership of ZarinPal's production writes without a new approved spec.
- Treat source `adjusted_fee` as the merchant's real fee for this product. Normalize it once per session as `fee_amount_rial`; do not sum the repeated attempt-row value.
- Treat both `Paid` and `Verified` as successful bank payments. `Paid` still requires merchant verification; `Verified` is finalized. Never classify `Paid` as failure.
- A source record with `try_seq = 0` and `try_status = NoAttempt` creates a session with zero attempts. Never insert it into `payment_attempts` or count it as a failed/retried attempt.
## Go baseline after approval
@@ -30,7 +31,7 @@ For every Go-related task, always invoke and follow the `golang-how-to` skill, t
- Use parameterized queries and transactions with explicit boundaries. Stream/bulk-load the multi-million-row input instead of holding it in memory.
- Use Ent for application persistence and relationship modeling. Use pgx with PostgreSQL `COPY` for the bulk-import staging path; do not force high-volume ingestion through row-by-row ORM operations.
- Database migrations are explicit, versioned, reviewed artifacts applied by a dedicated migration/deployment step. Never run Ent automatic schema creation against production during application startup.
- Passwords use an approved adaptive hash (Argon2id or bcrypt per the approved auth spec); sessions use secure, HttpOnly cookies and server-side revocation.
- Passwords use Argon2id with reviewed parameters and a unique salt. Sessions are server-side and revocable; browsers receive only an opaque token in a `Secure`, `HttpOnly`, `SameSite=Lax` cookie. Never put authentication tokens in browser storage.
- Required verification includes unit/integration tests, race detection where applicable, `go vet`, `golangci-lint`, `govulncheck`, and migration/import tests.
## Git and documentation

View File

@@ -8,17 +8,25 @@ A business represented in the source data. A merchant owns terminals, payment se
A person/account permitted to authenticate as one merchant. The hackathon seed creates an initial demo user for each merchant; a merchant and a merchant user are not the same concept.
## Authenticated Session
A time-bounded and revocable authorization grant connecting one authenticated merchant user to exactly one merchant boundary. Logout, account disablement, idle expiry, or absolute expiry ends the grant.
## Terminal
A merchant-owned payment endpoint identified by the source terminal key.
A merchant-owned payment endpoint identified by the source terminal key. In the challenge archive, the terminal also carries its invariant verification method (`Automated` or `Manual`); one merchant may own terminals configured differently.
## Payment Session
One payment journey identified by the source session key. A session belongs to one merchant and terminal and can contain one or more payment attempts.
One payment journey identified by the source session key. A session belongs to one merchant and terminal and can contain zero or more payment attempts.
## Payment Attempt
One try within a payment session, distinguished by its try sequence. This is the grain of a source CSV row and must not be confused with a unique payment session.
One actual try within a payment session, distinguished by a positive try sequence. Most source rows represent attempts, but a source row with `try_seq = 0` and `try_status = NoAttempt` is a session marker and must not become an attempt.
## No-Attempt Session
A payment session for which the source explicitly records `try_seq = 0` and `try_status = NoAttempt`. It has zero payment attempts; the marker preserves the session without inventing an attempt.
## Bank-Paid Session
@@ -34,11 +42,11 @@ A bank-paid session whose bank step succeeded but whose merchant verification st
## Merchant Fee
The Rial fee associated with one payment session. The challenge source calls it `adjusted_fee` and repeats the same value on each attempt row; the normalized model stores it once on the session as `fee_amount_rial`.
The product's real Rial fee associated with one payment session. It is counted once per session regardless of the number of attempts.
## Opaque Payer Token
The nullable `payer_card_key` source value. It is an unverified, merchant-scoped token used only for internal approximate repeat-payer analytics. It is not a card number or verified person identifier.
The nullable `payer_card_key` source value: a pseudonymous card identifier whose equality is meaningful only within one merchant. It supports internal repeat-card analytics where populated, but it is not a card number, a cross-merchant identifier, or a verified person identifier.
## Import Run
@@ -58,11 +66,11 @@ A deterministic numerical measure with a definition, source fields, filters, per
## Analytics Projection
A rebuildable, versioned analytics result derived from normalized PostgreSQL facts and cached in Redis for one merchant/cohort revision. It is not independent business truth.
A rebuildable, versioned analytics result derived from normalized PostgreSQL facts and cached in Redis for one immutable dataset snapshot. It is not independent business truth.
## Analytics Revision
## Dataset Snapshot
A monotonically increasing PostgreSQL merchant or cohort version changed when a related payment fact is inserted. Redis cache keys include the revisions they represent; a mismatch causes request-triggered section jobs to calculate new keys while older keys expire through TTL.
The one completed import run selected for the MVP. Its immutable import-run id, source checksum, and schema version identify the PostgreSQL facts represented by a Redis cache key. The MVP does not mutate payment facts or require cache-invalidation triggers.
## Analytics Section

18
Makefile Normal file
View File

@@ -0,0 +1,18 @@
.PHONY: test vet build migrate import seed api worker
test:
go test ./...
vet:
go vet ./...
build:
go build ./cmd/...
migrate:
go run ./cmd/migrate
import:
go run ./cmd/importer
seed:
go run ./cmd/admin
api:
go run ./cmd/api
worker:
go run ./cmd/worker

View File

@@ -2,7 +2,27 @@
Go backend for merchant identity, dataset ingestion, deterministic analytics, authenticated APIs, and later AvalAI integration.
Current state: specification bootstrap only. There is intentionally no Go module or application scaffold yet because the architecture and dependency-injection choices still require approval.
Current state: the local MVP implements database migration, streaming dataset import,
secure demo identity, deterministic dashboard analytics, merchant-scoped details/CSV,
the authenticated HTTP API, and a Redis-connected worker boundary.
## Local development
Copy `.env.example` to the ignored `.env`, load its variables in your shell, and run:
```text
go run ./cmd/migrate
go run ./cmd/importer
go run ./cmd/admin
go run ./cmd/worker
go run ./cmd/api
```
`cmd/admin` idempotently creates or resets `DEMO_USERNAME` using `DEMO_PASSWORD`
and assigns it to the imported merchant with the largest session population. It
never prints the password or stores it outside the Argon2id hash. The local API
uses the `zpa_session` HttpOnly cookie when `COOKIE_SECURE=false`; secure
deployments use the contract's `__Host-zpa_session` cookie.
## Local dataset input
@@ -12,7 +32,9 @@ Place the provided archive at:
data/incoming/challenge_data.csv.gz
```
The entire `data/incoming/` payload is ignored by Git. The one-time bootstrap importer will read from this location (or an explicitly supplied path), stream the compressed CSV, validate it, and write an import manifest containing the source checksum, schema version, row counts, quality results, and importer revision. It is an administrative backfill/recovery command, not a scheduled ingestion job. Future production payment data is populated in PostgreSQL by ZarinPal through an upstream contract outside this project's scope. The raw file and resulting PostgreSQL data are not committed.
The entire `data/incoming/` payload is ignored by Git. The one-time bootstrap importer will read from this location (or an explicitly supplied path), stream the compressed CSV, validate it, and write an import manifest containing the source checksum, schema version, row counts, quality results, and importer revision. It is an administrative bootstrap/recovery command, not a scheduled ingestion job. The hackathon MVP does not mutate payment facts after this import; any future ZarinPal live-write integration is outside this project's current scope. The raw file and resulting PostgreSQL data are not committed.
The proposed first-server Docker flow and Bash entrypoint are specified in [`docs/IMPORT-WORKFLOW-SPEC.md`](docs/IMPORT-WORKFLOW-SPEC.md). Implementation is authorized and must follow the dependency order in [`tasks/todo.md`](tasks/todo.md).
## Planned runnable boundaries
@@ -24,4 +46,9 @@ One repository may expose several small `cmd/` entry points while sharing domain
- HTTP API with authenticated SSE analytics delivery
- on-demand analytics worker that calculates dashboard sections in parallel and caches versioned results in Redis
The exact package layout is deferred until the Go architecture and DI decisions are approved.
The approved package-layout and manual-constructor decisions are implemented through the scaffold and provider tasks.
## Merchant API contract
- [`docs/API-SPEC.md`](docs/API-SPEC.md) defines the reviewable MVP capability, security boundaries, SSE recovery, detail pagination, and CSV-export policy.
- [`docs/openapi.yaml`](docs/openapi.yaml) is the approved OpenAPI 3.1 interface source of truth for generated frontend types.

40
cmd/admin/main.go Normal file
View File

@@ -0,0 +1,40 @@
package main
import (
"context"
"log/slog"
"os"
"time"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/config"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/database"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/identity"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("configuration failed", "error", err)
os.Exit(1)
}
if err := cfg.ValidateSeed(); err != nil {
logger.Error("seed configuration failed", "error", err)
os.Exit(1)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
pool, err := database.Open(ctx, cfg.DatabaseURL)
if err != nil {
logger.Error("database connection failed", "error", err)
os.Exit(1)
}
defer pool.Close()
result, err := identity.SeedDemo(ctx, pool, cfg.DemoUsername, cfg.DemoPassword)
if err != nil {
logger.Error("demo account seed failed", "error", err)
os.Exit(1)
}
logger.Info("demo account ready", "username", result.Username, "merchant_source_key", result.MerchantSourceKey,
"merchant_name", result.MerchantName, "password_source", "DEMO_PASSWORD")
}

51
cmd/api/main.go Normal file
View File

@@ -0,0 +1,51 @@
package main
import (
"context"
"errors"
"log/slog"
"net/http"
"os"
"os/signal"
"time"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/analytics"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/api"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/config"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/database"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/identity"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("configuration failed", "error", err)
os.Exit(1)
}
if err := cfg.ValidateAPI(); err != nil {
logger.Error("api configuration failed", "error", err)
os.Exit(1)
}
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
pool, err := database.Open(ctx, cfg.DatabaseURL)
if err != nil {
logger.Error("database connection failed", "error", err)
os.Exit(1)
}
defer pool.Close()
handler := api.NewServer(identity.New(pool), analytics.New(pool, cfg.CursorSecret), logger, cfg.FrontendOrigin, cfg.CookieSecure).Handler()
server := &http.Server{Addr: cfg.HTTPAddr, Handler: handler, ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 65 * time.Second, IdleTimeout: 60 * time.Second}
go func() {
<-ctx.Done()
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
defer shutdownCancel()
_ = server.Shutdown(shutdownCtx)
}()
logger.Info("api listening", "address", cfg.HTTPAddr)
if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
logger.Error("api failed", "error", err)
os.Exit(1)
}
}

36
cmd/importer/main.go Normal file
View File

@@ -0,0 +1,36 @@
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/config"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/database"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/importer"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("configuration failed", "error", err)
os.Exit(1)
}
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
pool, err := database.Open(ctx, cfg.DatabaseURL)
if err != nil {
logger.Error("database connection failed", "error", err)
os.Exit(1)
}
defer pool.Close()
result, err := importer.Run(ctx, pool, cfg.ImportPath)
if err != nil {
logger.Error("import failed", "error", err)
os.Exit(1)
}
logger.Info("import complete", "noop", result.IsNoop, "source_sha256", result.SourceSHA256,
"source_rows", result.SourceRows, "session_rows", result.SessionRows, "attempt_rows", result.AttemptRows)
}

38
cmd/migrate/main.go Normal file
View File

@@ -0,0 +1,38 @@
package main
import (
"context"
"log/slog"
"os"
"time"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/config"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/database"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("configuration failed", "error", err)
os.Exit(1)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
pool, err := database.Open(ctx, cfg.DatabaseURL)
if err != nil {
logger.Error("database connection failed", "error", err)
os.Exit(1)
}
defer pool.Close()
migration, err := os.ReadFile("migrations/000001_mvp.sql")
if err != nil {
logger.Error("reading migration failed", "error", err)
os.Exit(1)
}
if err := database.Migrate(ctx, pool, string(migration)); err != nil {
logger.Error("migration failed", "error", err)
os.Exit(1)
}
logger.Info("migration complete", "version", 1)
}

33
cmd/worker/main.go Normal file
View File

@@ -0,0 +1,33 @@
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"time"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/config"
"github.com/redis/go-redis/v9"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("configuration failed", "error", err)
os.Exit(1)
}
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
client := redis.NewClient(&redis.Options{Addr: cfg.RedisAddr, Password: cfg.RedisPassword})
defer client.Close()
pingCtx, pingCancel := context.WithTimeout(ctx, 5*time.Second)
defer pingCancel()
if err := client.Ping(pingCtx).Err(); err != nil {
logger.Error("redis connection failed", "error", err)
os.Exit(1)
}
logger.Info("analytics worker ready", "redis_address", cfg.RedisAddr)
<-ctx.Done()
}

View File

@@ -2,6 +2,8 @@
Copy the challenge dataset to this directory as `challenge_data.csv.gz` before running the future importer. The archive is intentionally ignored by Git.
The server-side Docker/bootstrap contract is documented in [`docs/IMPORT-WORKFLOW-SPEC.md`](../../docs/IMPORT-WORKFLOW-SPEC.md). During deployment implementation, the operator places the same filename under the deployment repository's Git-ignored `data/incoming/` directory; the first-start Bash script mounts it read-only into the one-shot importer container.
Expected source columns:
```text
@@ -14,3 +16,4 @@ verified_at, settled_at, expire_in
Do not add merchant passwords or other secrets to this directory. Demo credentials will be generated/seeding inputs managed separately through secure environment or one-off administrative workflows.
The source contains session markers where `try_seq = 0` and `try_status = NoAttempt`. The importer must create the session but no normalized payment-attempt row for these records.

183
docs/ANALYTICS-SPEC.md Normal file
View File

@@ -0,0 +1,183 @@
# Analytics Engine Capability Spec
- Status: Approved specification — implementation authorized 2026-08-20
- Capability: Analytics engine
- Depends on: Approved Data Foundation specification
- Last updated: 2026-08-20
## Objective
Turn normalized ZarinPal payment facts into trustworthy, merchant-scoped analytics that explain payment performance, verification gaps, retry outcomes, processing time, fees, trends, and same-category position. Deterministic calculations are the product source of truth; later AI assistance may explain or recommend actions from sanitized aggregates but cannot define or alter metrics.
## Actors and consumers
- Authenticated merchant: sees detailed metrics and drill-downs only for its merchant boundary.
- Merchant API: requests versioned analytics sections and exposes typed results/events.
- Dashboard: renders sections independently as they become ready.
- Analytics worker: calculates idempotent section results from PostgreSQL and caches approved aggregates in Redis.
- Future AI adapter: post-MVP consumer of minimized, approved aggregate facts only.
## In scope
- Daily, Saturday–Friday weekly, Jalali-month, and Persian-season analytics in `Asia/Tehran`
- Overview, payment performance, fees, and peer-comparison sections
- Payment success and volume with correct `Paid`/`Verified` semantics
- Verification completion and unverified-paid gaps
- Retry recovery and attempt/session diagnostics
- Processing-time p50/p95 with sample and coverage
- Real merchant fee amounts and paid fee rate at session grain
- Current-period trends against an approved equivalent baseline
- Same-category merchant-weighted peer median and performance percentile
- Lazy, bounded parallel calculation with partial SSE delivery
- Versioned 30-day Redis cache keyed by the immutable MVP dataset snapshot
- Traceability from aggregate metrics to permitted merchant-owned evidence
- Merchant-wide or authenticated merchant-terminal scope for deterministic merchant metrics
- Bounded daily payment-history points and filtered read-only transaction evidence
## Out of scope for MVP
- Holiday or named-event reporting
- AI-generated recommendations or free-form LLM analysis
- Forecasting, causal claims, or automated campaign execution
- Arbitrary merchant-created formulas or cross-merchant filters
- Real-time streaming payment ingestion owned by this service
- Persisting calculated analytics as PostgreSQL business facts
- Public or unauthenticated analytics
## Canonical sections
| Section id | Responsibility | Minimum result |
|---|---|---|
| `overview` | Merchant-level outcome and trend summary | bank-paid sessions/amount, success rate, verification completion, selected period, prior-period comparison |
| `payment_performance` | Status, retry, and timing analysis | paid/verified/unverified/failed/reversed counts, retry recovery, init/verify p50/p95 with coverage |
| `fees` | Merchant fee analysis | paid fee amount, paid amount denominator, paid fee rate, period trend |
| `peer_comparison` | Privacy-bounded same-category benchmark | eligible peer count/session count, cohort median, direction-aware merchant percentile or typed suppression |
Every section uses a versioned typed contract and has independent `queued`, `calculating`, `ready`, `failed`, and where applicable `suppressed` states.
## Metric and period contract
- `METRIC-CATALOGUE.md` version `0.4.0` is canonical for formulas, units, grains, missing-data behavior, and peer direction.
- `Paid` and `Verified` both mean bank payment succeeded; only `Verified` is finalized. Never label `Paid` as failure.
- Amount and fee calculations count each eligible payment session once. Attempt rows cannot multiply session values.
- Rates are calculated from summed numerator and denominator components for the selected period, never averaged from daily percentages.
- Timing percentiles exclude null values and expose sample size and non-null coverage; missing time never becomes zero.
- `NoAttempt` markers create zero-attempt sessions and never count as attempts or retries. `InBank` is an observed attempt stage, not proof of rejection.
- Every result states the exact Tehran-local half-open period, Jalali label, unit, sample/denominator, dataset snapshot, metric-definition version, and calculation timestamp.
- Approved period types are one Tehran-local day, one Saturday–Friday week, one Jalali month, or one Persian season. Holiday/event dimensions do not exist in the MVP contract.
## Peer comparison contract
- Eligible peers are other merchants in the active merchant's category for the same selected period.
- Suppress the peer result unless at least 10 other merchants and 1,000 combined peer sessions contribute.
- Calculate each peer merchant's metric independently and take the median of those merchant values; do not transaction-weight the cohort.
- Return the cohort median and active merchant's performance percentile. Higher displayed percentile always means better, so processing-time and fee-rate ranking is inverted.
- Benchmark payment success, verification completion, retry recovery, init/verify p50/p95, and paid fee rate.
- Never return peer identities, individual values/ranks, rows, or arbitrary filters that enable differencing attacks.
- Peer comparison remains merchant-grain. A terminal-scoped run suppresses this section with `TERMINAL_SCOPE_UNSUPPORTED` instead of presenting merchant-wide peer values as terminal-scoped evidence.
## Terminal scope and payment history
- `terminalId` is optional and is resolved only under the merchant from the authenticated server-side session. Omission means all merchant terminals; unknown and other-merchant identifiers fail identically.
- Overview, payment-performance, fees, and daily payment-history calculations use the selected terminal predicate when present.
- Daily payment-history points use Tehran-local dates and return total sessions, bank-paid sessions, and bank-paid amount in Rial for each date with source sessions in the selected period/scope.
- The payment-history array is bounded by the maximum approved period length and never contains payer tokens, peer rows, or attempt-grain money.
## Transaction evidence
- The `ALL`, `UNVERIFIED_PAID`, `FAILED`, and `RETRIED` views are scoped by authenticated merchant and approved period.
- Optional terminal, status, failure-stage, and non-negative amount bounds are allowlisted and parameterized. Failure stage applies only to `FAILED`.
- Keyset cursors are authenticated-encrypted, bound to merchant/period/canonical filters, and expire after 24 hours; replay under another binding is rejected.
- One-session detail resolves the opaque session id under the merchant and returns ordered attempts without payer-card keys or response-code fields.
- JSON and CSV reuse the same canonical predicate. CSV omits cursor/page limit, is capped at 100,000 rows, and applies a 60-second deadline.
## Calculation and cache lifecycle
1. An idempotent authenticated start request resolves merchant, selected period, immutable dataset snapshot, and metric-definition version.
2. Equivalent requests map to one current analytics run and do not enqueue duplicate work.
3. Missing section results are enqueued together and calculated independently by the Go worker through Asynq `v0.26.0`, explicitly a pre-1.0 dependency pinned for compatibility.
4. Worker limits default to 16 jobs globally and four per merchant. All four core sections begin in bounded parallel and become visible independently.
5. Each attempt times out after 60 seconds. Transient failures receive at most two retries using exponential backoff of approximately one then two seconds plus jitter; deterministic failures are not retried.
6. Successful section results are published atomically to Redis under keys containing the immutable dataset snapshot, merchant, period, section, and metric-definition version, with a 30-day TTL. PostgreSQL normalized facts remain authoritative.
7. Payment facts do not change during the hackathon MVP, so no insert/update/delete invalidation trigger exists. Any future dataset replacement requires a separately approved recovery and cache-namespace-selection contract.
8. Redis loss causes safe lazy recomputation. It never causes loss of payment facts or metric definitions.
## Partial delivery contract
- A separate authenticated SSE connection streams run/section state and each completed section immediately.
- SSE sends a heartbeat every 15 seconds and retains authorized replay events for one hour.
- Reconnect uses the last event id where retained; otherwise the client fetches the current run snapshot and reopens the stream.
- One failed section never hides ready siblings. Manual retry targets only an exhausted failed section and cannot clear successful results.
- Start limits are 10 requests/minute with burst 3; manual retry limits are 3 requests/10 minutes with burst 1; SSE limits are two concurrent connections/user and four/merchant. Independent user and merchant buckets return `429` with `Retry-After` before work is queued.
## Traceability and privacy
- Merchant-owned aggregate cards link to definitions and, where approved, tenant-scoped session/attempt evidence that reconciles to the aggregate.
- Payer-card tokens remain internal, opaque, merchant-scoped inputs and never appear in APIs, logs, exports, caches, peer results, or AI payloads.
- Redis values contain approved aggregates and trace metadata only, not raw payment rows or credentials.
- Cross-merchant calculations execute server-side and return only approved aggregate/suppressed contracts.
- Future AvalAI prompts may receive minimized aggregates only after a separate AI capability approval.
## Failure and edge behavior
- Zero, undefined denominator, unavailable source, peer suppression, calculation failure, and not-yet-ready are distinct typed states.
- Unknown future payment statuses are excluded from success numerator/denominator and reported through coverage evidence until classified.
- `Reversed` remains outside success and is displayed separately when non-zero.
- Empty approved periods return a valid empty result, not fabricated zero rates or a system failure.
- Section publication is atomic; partial/failed calculations never overwrite a valid ready cache entry.
- Database, Redis, queue, timeout, and authorization errors are distinguishable internally while merchant-facing errors remain safe and actionable.
## Actionable insight threshold
- A deterministic actionable insight is eligible only when its comparison contains at least 100 relevant payment sessions.
- Rate insights require an absolute change of at least 3 percentage points. Amount and count insights require an absolute relative change of at least 20 percent.
- The threshold is a display eligibility rule, not a change to the underlying metric. Exact metrics and comparisons remain available even when no insight card is emitted.
- A zero, missing, or undefined baseline cannot produce a relative-change insight. Return the metric and an explicit unavailable comparison instead of dividing by zero or inventing a trend.
- Meeting a threshold does not prove causation. Insight language describes the observed change and a possible merchant action without claiming why it happened.
## Observability
- Measure section queue delay, calculation duration, query duration, cache hit/miss, retry count, failure class, SSE delivery/reconnect, and suppression rate.
- Correlate start request, analytics run, section job, cache key version, and SSE event without logging sensitive merchant rows or payer tokens.
- Record PostgreSQL query plans during performance verification and add indexes only from measured access paths.
- Define service-level objectives and alert thresholds in the Platform Operations spec, not implicitly in worker code.
## Acceptance criteria
- Every metric reconciles to an independent PostgreSQL query over normalized facts for representative merchants and all approved period types.
- Session-grain amount/fee totals remain unchanged when a session has multiple attempts.
- `Paid` plus `Verified` success, verification completion, retry recovery, and timing coverage match catalogue definitions.
- Saturday–Friday, Jalali-month, and Persian-season boundaries match `calendar_dates` and API half-open ranges.
- First access starts all missing sections once; ready sections stream independently and equivalent concurrent starts deduplicate.
- Cache hits avoid repeating source aggregation; the immutable completed-import snapshot keeps equivalent requests in one stable namespace.
- Redis flush causes correct cold-cache rebuilding without source-data loss.
- Timeouts, transient retry limits/backoff, failed-section isolation, manual retry, rate limits, heartbeat, replay, and snapshot fallback behave as specified.
- Peer results exclude the active merchant, use equal merchant weighting, obey both suppression thresholds, and never expose individual peers.
- Tenant-bound drill-downs reconcile to aggregates and cannot expose another merchant through parameter manipulation or cursor replay.
- Read-only drill-downs cover all, unverified-paid, failed, and retried sessions. Failed-session filters may distinguish zero-attempt and reached-bank stages without exposing payer tokens or claiming undocumented causes.
- No raw payer token, credential, cross-tenant row, or AvalAI secret enters results, logs, cache payloads, or browser data.
## Boundaries
- Always: use catalogue definitions, preserve grain, enforce tenant scope, expose period/unit/coverage/version, calculate idempotently, and keep results reproducible.
- Ask first: add/change a metric, period, peer cohort, threshold, cache key/version, drill-down, AI input, or cross-service interface.
- Never: average rates, multiply session money by attempts, classify `Paid` as failure, expose peer rows, trust client tenant ids, or let AI become the metric source of truth.
## Resolved decisions
- Normalized PostgreSQL facts are authoritative; Redis is a 30-day disposable analytics cache.
- Four sections calculate lazily and in bounded parallel, with partial authenticated SSE delivery.
- Reporting covers every complete day, Iranian week, Jalali month, and Persian season inside the imported dataset's local-date coverage; arbitrary client-computed ranges remain disallowed.
- Peer comparison uses same-category merchant-weighted medians and direction-aware percentiles with the approved suppression threshold.
- Payment lifecycle, retry, timing, and real-fee definitions follow `METRIC-CATALOGUE.md` and ADR 0013.
- Asynq `v0.26.0` is pinned and acknowledged as pre-1.0; jobs have bounded concurrency, timeouts, rate limits, and exponential-backoff retries.
- Default first view is the latest complete Jalali month containing source data, avoiding an empty current month in the static demo.
- Compare with the immediately previous equivalent complete period. If it is empty or unavailable, return an explicit unavailable comparison rather than silently substituting another baseline.
- Overview priority is bank-paid amount, bank-paid session count, payment success rate, then verification completion. Retry, timing, and fee analysis follow below.
- MVP read-only drill-downs cover all, unverified-paid, failed, and retried sessions; payer tokens and peer-level rows are excluded.
- Terminal-scoped merchant metrics are supported, while merchant-grain peer comparison is explicitly suppressed for terminal runs.
- Actionable insight cards require at least 100 relevant sessions and either a 3-percentage-point rate change or a 20-percent amount/count change. Below-threshold comparisons remain visible as metrics but do not generate an action claim.
## Open product decisions before implementation planning
- None. Any new or changed metric, period, cohort, threshold, or cross-service contract requires a specification amendment before implementation.

298
docs/API-SPEC.md Normal file
View File

@@ -0,0 +1,298 @@
# Merchant Analytics API Capability Spec
- Status: Approved specification — implementation authorized 2026-08-20
- Capability: Merchant API
- Depends on: Approved Identity, Data Foundation, and Analytics Engine specifications
- Interface source of truth: `docs/openapi.yaml`
- Last updated: 2026-08-20
## Objective
Expose the approved merchant identity and analytics capabilities as a tenant-safe REST JSON API under `/api/v1`. The API starts and observes analytics runs, streams partial results, lists merchant-owned evidence, and exports the same authorized detail view as CSV. OpenAPI is the normative interface contract and is intended to generate frontend types after approval.
## Actors and trust boundaries
- Unauthenticated browser: may call login and explicitly public health/static resources only.
- Authenticated merchant user: may access one merchant derived from the current server-side session.
- Merchant dashboard: same-origin browser client using the opaque session cookie and CSRF protection for mutations.
- Analytics worker and Redis: private implementation boundaries; neither is exposed through this API.
- Cross-merchant peer data: server-only calculation input. The API returns only approved anonymized aggregates or a suppressed state.
Every request, query parameter, cursor, run id, event id, and export request is untrusted. Possession of a resource id never grants access. The backend resolves the merchant from `__Host-zpa_session` for every protected request and checks resource ownership inside that boundary.
## Scope
### In scope for MVP
- Login, current-session inspection, and logout
- Approved reporting-period discovery with exact ISO boundaries and Persian/Jalali labels
- Idempotent analytics-run start, snapshot retrieval, section retry, and authenticated SSE
- Typed results for `overview`, `payment_performance`, `fees`, and `peer_comparison`
- Cursor-paginated, read-only merchant-owned session details for unverified-paid, failed, and retried views
- Synchronous CSV export of the currently authorized detail view and filters
- RFC 9457 Problem Details, authorization, throttling, response-size limits, audit, and observability contracts
### Out of scope for MVP
- Merchant or user administration, registration, password recovery, and browser operator access
- Arbitrary formulas, date ranges, filters, sorting, cross-merchant drill-down, and peer-row export
- Payer-token, credential, session-token, raw cache/job, or AI interfaces
- Asynchronous export jobs, saved reports, scheduled delivery, and export history
- Production payment writes, CSV ingestion, migrations, deployment endpoints, and AvalAI calls
## Protocol conventions
- HTTPS is required outside explicit local development. Successful structured responses use `application/json`; errors use `application/problem+json`.
- JSON field and query names use `camelCase`; enum values use `UPPER_SNAKE_CASE`; canonical metric and section ids retain their approved `snake_case` values.
- Canonical timestamps are ISO 8601/RFC 3339 instants in UTC with `Z`. Period objects also return their Tehran-local half-open boundaries with explicit `+03:30` offsets and a ready-to-display Persian/Jalali label.
- Money is an integer number of Rial. Rates and percentiles are fixed-precision decimal JSON strings, never binary floating-point values.
- Unknown JSON fields and unknown query parameters are rejected with `VALIDATION_ERROR` so misspelled filters cannot silently broaden a result.
- Protected responses use `Cache-Control: private, no-store`. Responses include `X-Request-ID`; a safe client-supplied request id may be accepted but never trusted as unique.
- State-changing browser requests require both the approved CSRF token header and strict same-origin evidence. Login is unauthenticated but still enforces origin checks and its approved throttles.
## Resource map
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/v1/auth/sessions` | Authenticate and create a server-side session |
| `GET` | `/api/v1/auth/session` | Return the current merchant-user session summary |
| `DELETE` | `/api/v1/auth/session` | Revoke the current session and clear its cookie |
| `GET` | `/api/v1/analytics/periods` | List approved period choices and the default period |
| `GET` | `/api/v1/analytics/terminals` | List terminals owned by the authenticated merchant |
| `POST` | `/api/v1/analytics/runs` | Idempotently create or return the current run |
| `GET` | `/api/v1/analytics/runs/{runId}` | Return the current run/section snapshot |
| `GET` | `/api/v1/analytics/runs/{runId}/events` | Stream authorized SSE run events |
| `POST` | `/api/v1/analytics/runs/{runId}/sections/{sectionId}/retry-attempts` | Retry one exhausted failed section |
| `GET` | `/api/v1/analytics/session-details` | List authorized merchant-owned session evidence |
| `GET` | `/api/v1/analytics/sessions/{sessionId}` | Return one merchant-owned session and ordered attempts |
| `GET` | `/api/v1/analytics/session-details.csv` | Export the same authorized detail view as CSV |
The request/response schemas, required headers, status codes, examples, and stable `operationId` values are defined in `docs/openapi.yaml`.
## Reporting-period contract
Clients select one allowlisted `periodId` returned by `GET /analytics/periods`; they do not submit arbitrary start/end timestamps. Each period contains:
- stable `periodId` and `periodType` (`DAY`, `WEEK`, `JALALI_MONTH`, or `PERSIAN_SEASON`);
- exact half-open UTC query boundaries `startAt` and `endAt`;
- exact Tehran-local boundaries `localStartAt` and `localEndAt`;
- `timezone: Asia/Tehran` and `jalaliLabelFa`, ready for display;
- `isComplete` and `isDefault` flags.
The catalog contains every complete day, Saturday–Friday week, Jalali month, and Persian season inside the imported dataset's local-date coverage. Descriptors are grouped by type and ordered newest-first within each group. The default is the latest complete Jalali month containing source data. Weekly periods start Saturday at `00:00` and end at the following Saturday `00:00` in Tehran. The optional `periodType` query is allowlisted and filters the returned descriptors. The server rejects stale or fabricated period ids and invalid period types instead of accepting client-computed dates.
## Analytics-run contract
`POST /analytics/runs` accepts `periodId`, an optional merchant-owned `terminalId`, and an `Idempotency-Key` representing one dashboard-start intent. An omitted terminal selects the whole merchant. The server resolves a supplied terminal under the session merchant before calculation and binds the key to the authenticated user, merchant, canonical request, immutable dataset snapshot, and metric-definition version.
- The same key and same canonical request replays the original status/body without duplicate jobs.
- The same key with a different request returns `422 IDEMPOTENCY_KEY_REUSED`.
- Equivalent concurrent requests for the current dataset snapshot resolve to the same run even when their valid intent keys differ.
- An in-flight duplicate returns the current `202` run snapshot rather than starting another effect.
- Intent records outlive the one-hour event replay window and the maximum job retry chain; the concrete retention value is an implementation parameter that must be documented before implementation.
A run snapshot identifies the selected period, optional terminal scope, immutable dataset snapshot, metric-definition version, creation/update times, overall state, and all four sections. Section states are `QUEUED`, `CALCULATING`, `READY`, `FAILED`, or `SUPPRESSED`. A failure includes only a safe stable error code and retry eligibility. Ready section payloads use the typed schemas in OpenAPI. Because the approved peer cohort is merchant-grain, terminal-scoped runs return peer comparison as `SUPPRESSED` with `TERMINAL_SCOPE_UNSUPPORTED`; they never place merchant-wide peer values under a terminal scope.
## Metric response rules
- Every metric value identifies its canonical metric id, unit, definition version, sample/denominator or coverage where applicable, and calculation time through its containing section.
- Count and money values are JSON integers. Percentage, ratio, and percentile values are decimal strings or `null` when mathematically undefined.
- Zero, undefined denominator, unavailable comparison, suppressed peer result, not-yet-ready, and calculation failure are distinct states.
- `Paid` and `Verified` are bank-paid; only `Verified` is finalized. Session money and fees are counted once regardless of attempt count.
- Timing values include p50, p95, contributing sample size, and coverage. Missing timings never become zero.
- Peer outputs include only the approved merchant-weighted cohort median and direction-adjusted merchant percentile after both suppression thresholds pass.
## Detail views and cursor pagination
`GET /analytics/session-details` requires an approved `view`:
- `ALL`: all merchant-owned sessions in the selected scope;
- `UNVERIFIED_PAID`: sessions currently in `Paid`;
- `FAILED`: sessions currently in `Failed`, optionally filtered by `failureStage=NO_ATTEMPT|REACHED_BANK|OTHER_OBSERVED`;
- `RETRIED`: sessions with more than one normalized attempt.
All views require one approved `periodId`. Optional `terminalId`, `status`, `failureStage`, `minAmountRial`, and `maxAmountRial` filters are validated inside the merchant boundary. The failure-stage filter is valid only for `FAILED`, and amount bounds must be non-negative and ascending. There is no merchant-id, peer, payer-token, free-text, response-code, PSP, bank, or arbitrary date filter in the MVP.
Ordering is stable and deterministic: `createdAt DESC, sessionId DESC`. The authenticated-encrypted cursor binds the merchant, period, view, canonical filters, ordering, and last row, expires after 24 hours, and reveals no internal ids or filter data when decoded by a client. The encryption key comes from the independently configured `CURSOR_SECRET` and is never returned or logged.
- `limit` defaults to 25 and is limited to 1–100.
- Responses contain `items`, `pageInfo.limit`, `pageInfo.hasNextPage`, and nullable `pageInfo.nextCursor`.
- Total counts are omitted from page responses to avoid an additional unbounded count query. Aggregate cards remain the reconciliation source.
- Reusing a cursor with changed filters, another merchant, or after expiry returns `400 INVALID_CURSOR` without disclosing which binding failed.
Detail rows expose only fields available in the authorized view: opaque merchant-local `sessionId`, the existing opaque source terminal reference labeled `شناسه پایانه`, status, amount/fee in Rial, timestamps, attempt count, and a bounded ordered attempt summary where relevant. The terminal reference is resolved and displayed only inside the authenticated merchant boundary and never grants authorization. Rows exclude source merchant keys, payer tokens, credentials, raw session tokens, peer data, internal database keys, and undocumented response-code meanings.
`GET /analytics/sessions/{sessionId}` applies the same session-derived merchant predicate before resolving the opaque session id. Missing and other-merchant ids return the same safe `404`; successful responses include ordered attempt summaries but never payer tokens or response-code fields.
## CSV export
`GET /analytics/session-details.csv` accepts exactly the same approved view and filters as the JSON detail endpoint, except cursor and page limit. The server reauthorizes and re-resolves the merchant at export time. It executes one immutable logical snapshot so rows do not drift between authorization, limit validation, and streaming.
### Format and delivery
- `Content-Type: text/csv; charset=utf-8`
- UTF-8 with one leading BOM for common Persian spreadsheet compatibility
- RFC 4180-style comma separation, header row, CRLF records, and doubled quotes
- Stable Persian column names separated by normal spaces, with Persian status labels
- UTC ISO timestamps with `Z`; Rial integer values; no locale-formatted numbers
- `Content-Disposition: attachment` with an ASCII `filename` fallback followed by an RFC 6266 `filename*=UTF-8''...` value
- `Cache-Control: private, no-store` and `X-Content-Type-Options: nosniff`
The filename contains the view, canonical period id, and generation timestamp. It never contains a username, merchant name/key, raw filter text, or other user-controlled value.
### Safeguards
- Export columns are an allowlisted subset of the equivalent authorized JSON detail view. A CSV never adds payer tokens, credentials, peer-level data, internal ids, or fields unavailable in that view.
- Text cells that begin with `=`, `+`, `-`, `@`, tab, CR, or LF after leading whitespace are prefixed with a single apostrophe before CSV quoting to prevent spreadsheet formula execution. Numeric money/count columns are emitted only after strict numeric typing.
- Embedded CR/LF and quotes are encoded per CSV rules; control characters not required by approved fields are rejected or replaced with a documented safe marker.
- The export fails before sending CSV bytes if its preflight result exceeds either 100,000 rows or 50 MiB estimated uncompressed output. It returns `413 EXPORT_TOO_LARGE` and instructs the merchant to narrow approved filters.
- Export generation has a 60-second server deadline. A timeout returns a Problem Details response if headers are unsent; a connection terminated after streaming begins is incomplete and must never be presented as a successful audited export.
- Export requests default to 3 per 10 minutes with burst 1 in independent authenticated-user and merchant buckets. A rejection returns `429` with `Retry-After` before query/stream work begins.
The row/byte/deadline/export-throttle values are approved MVP policy.
## SSE events and recovery
The run stream uses `text/event-stream; charset=utf-8`, disables intermediary buffering/compression, sends `Cache-Control: no-cache, no-transform`, and reauthorizes the session for connection and replay. Event ids are opaque, stable within the retained run history, and scoped to the run and merchant.
| Event | Payload purpose |
|---|---|
| `run.snapshot` | Full current snapshot on initial connection or replay reset |
| `section.started` | Section entered `CALCULATING` |
| `section.ready` | Typed ready section payload |
| `section.failed` | Safe failure code and manual-retry eligibility |
| `section.suppressed` | Peer section suppressed with threshold evidence only |
| `run.completed` | Terminal run snapshot after every section is terminal |
| `heartbeat` | Server time and run id; no analytics payload |
Each non-comment event contains `id`, `event`, and one-line JSON `data`. Heartbeats are named events every 15 seconds so clients can observe liveness. The server may send SSE comment lines additionally, but clients must not depend on them.
On reconnect, the browser sends `Last-Event-ID`. If the id remains within the one-hour authorized replay history, the server replays only later events in original order and then continues live. If it is absent, invalid, belongs elsewhere, or has expired, the stream emits a fresh `run.snapshot` with `recovery: RESET_TO_SNAPSHOT` and continues from new events; it never reveals whether an id belongs to another merchant. Session expiry/revocation terminates recovery access. A completed run emits its current snapshot and terminal event, then may close.
## Error contract
Every non-SSE error response uses RFC 9457 `application/problem+json` with:
```json
{
"type": "https://zarinpal.amiirkhl.ir/docs/errors/validation-error",
"title": "Request validation failed",
"status": 422,
"detail": "One or more request values are invalid.",
"instance": "/api/v1/analytics/runs",
"code": "VALIDATION_ERROR",
"requestId": "01J...",
"errors": [{ "field": "periodId", "reason": "UNKNOWN_PERIOD" }]
}
```
`code` is the stable application discriminator; clients must not parse `title` or `detail`. `requestId` is safe correlation metadata. Validation `errors` use allowlisted field names/reasons and never echo credentials, cookies, cursor contents, SQL, stack traces, peer identities, or sensitive request values.
| Status | Stable codes (non-exhaustive) |
|---|---|
| `400` | `MALFORMED_REQUEST`, `INVALID_CURSOR` |
| `401` | `UNAUTHENTICATED` |
| `403` | `FORBIDDEN`, `CSRF_FAILED` |
| `404` | `RESOURCE_NOT_FOUND` |
| `409` | `SECTION_NOT_RETRYABLE`, `RUN_REVISION_STALE` |
| `413` | `REQUEST_TOO_LARGE`, `EXPORT_TOO_LARGE` |
| `415` | `UNSUPPORTED_MEDIA_TYPE` |
| `422` | `VALIDATION_ERROR`, `IDEMPOTENCY_KEY_REUSED`, `UNKNOWN_PERIOD` |
| `429` | `RATE_LIMITED`, `CONNECTION_LIMIT_REACHED` |
| `500` | `INTERNAL_ERROR` |
| `503` | `DEPENDENCY_UNAVAILABLE`, `ANALYTICS_UNAVAILABLE` |
| `504` | `ANALYTICS_TIMEOUT`, `EXPORT_TIMEOUT` |
For tenant-owned ids, an absent resource and a resource outside the authenticated merchant boundary both return the same `404 RESOURCE_NOT_FOUND`. `429` and retryable `503` responses include `Retry-After` when a meaningful retry time is known.
## Authorization and privacy matrix
| Surface | Authentication | Authorization/privacy rule |
|---|---|---|
| Login | None; origin checked | Generic credential outcome and approved username/IP throttles |
| Current session/logout | Valid current session | Current user/session only; logout requires CSRF and origin evidence |
| Periods | Valid current session | Approved global period definitions; no merchant ids accepted |
| Run/snapshot/SSE/retry | Valid current session | Run merchant must equal session merchant; ids never authorize |
| Detail JSON/CSV | Valid current session | Query is built inside session merchant scope; export is an allowlisted projection |
| Peer result | Valid current session | Aggregate or suppression only; never peer rows/ids/exports |
The API never serializes `payer_card_key`, password hashes, raw session/CSRF tokens, cookie values, credential material, source merchant keys, individual peer values/ranks, Redis keys, job payloads, or AI secrets.
## Rate and size limits
Approved existing limits remain unchanged:
- login: five failures/normalized username and 30 failures/source IP per five minutes;
- analytics start: 10/minute, burst 3, independently per user and merchant;
- failed-section retry: 3/10 minutes, burst 1, independently per user and merchant;
- SSE: two concurrent streams/user and four/merchant;
- section attempt: 60 seconds and at most two transient retries.
API boundary limits:
- Approved: JSON request bodies are limited to 16 KiB; requests with no defined body reject non-empty bodies.
- Approved: the request target/query string is limited to 8 KiB at the public edge.
- Approved: detail pages default to 25 rows and cannot exceed 100.
- Approved: synchronous CSV export is limited to 100,000 rows, 50 MiB estimated uncompressed output, and 60 seconds, with 3 requests/10 minutes and burst 1 per user and merchant.
Limits are enforced before enqueuing or expensive work where possible. User and merchant buckets are independent; exhausting one never consumes another tenant's allowance.
## Audit and observability
### Audit events
Record login success/failure category, logout, analytics start/deduplication, manual retry, SSE authorization/replay reset, detail access, and CSV export requested/completed/rejected/aborted. Export audit records include user id, merchant id, canonical view/filter fingerprint, period id, row count, byte count, result, request id, and timestamps—not exported row values, payer tokens, credentials, raw cursors, filenames from users, or peer data.
Minimal demo authentication audit retention remains 30 days. Analytics-access and CSV-export audit records are retained for 90 days, are accessible only to explicitly authorized backend operators, and are automatically deleted after the retention window. The MVP has no merchant-facing audit endpoint or audit-history UI.
### Metrics and traces
Measure request duration/status/code/response size; rate-limit rejections by operation/bucket type; run deduplication; section queue/calculation/cache/retry behavior; SSE open duration, heartbeat, replay/reset, disconnect; detail query duration/page size; and export preflight/query/stream duration, rows, bytes, rejection/abort reason.
Correlate `requestId`, authenticated user/merchant internal ids, run id, section id, job id, and event id in structured telemetry. Do not place usernames, source ids, session ids from payment data, raw filter values, cookies, CSRF values, payer tokens, or merchant row payloads in metric labels or traces.
## Acceptance criteria
- OpenAPI validates as 3.1 and defines every operation, parameter, body, response, security requirement, SSE event, CSV header, and RFC 9457 problem schema named by this spec.
- Generated frontend types can consume JSON schemas without handwritten duplicate DTOs; all `operationId` values and schema names are stable and unique.
- Every protected endpoint resolves the active merchant from the server-side session and produces the same `404` for absent and cross-tenant resource ids.
- Login/logout/session behavior, cookie flags, CSRF/origin checks, expiry, revocation, and throttling match the approved Identity spec.
- Period discovery returns exact UTC and Tehran-local half-open boundaries plus a ready-to-display Persian/Jalali label; arbitrary ranges are rejected.
- Equivalent analytics starts deduplicate, idempotency-key payload conflicts fail, and all four section states/results match the Analytics spec and metric catalogue.
- SSE delivers independently completed sections, 15-second heartbeats, ordered one-hour replay, snapshot reset after replay loss, sibling failure isolation, and reauthorization after session changes.
- Detail pagination is deterministic, defaults to 25, caps at 100, and rejects a cursor reused across filters or tenant boundaries.
- Each detail view reconciles with its approved aggregate and exposes no payer token, credential, peer row, undocumented cause, or field unavailable to that view.
- CSV applies the same tenant/filter predicate and allowlisted projection as JSON detail, uses the documented UTF-8/CSV/content-disposition contract, neutralizes spreadsheet formulas, and emits no partial success audit on interruption.
- Request/rate/export limits reject before expensive work and return Problem Details plus `Retry-After` where applicable.
- Problem responses keep HTTP status and `status` aligned, expose stable `code`, and leak no stack, SQL, secret, cursor content, peer identity, or cross-tenant existence.
- Contract/security tests cover schema examples, unknown fields/params, cross-tenant ids and cursors, stale event ids, idempotency races, export filter parity, formula injection, row/byte/deadline caps, and sensitive-field absence.
- Analytics-access and CSV-export audit records are visible only to authorized backend operators and are deleted after 90 days.
## Boundaries
- Always: treat OpenAPI as source of truth; derive tenant scope from the session; validate inputs at the HTTP boundary; preserve approved metric/period semantics; use RFC 9457; paginate detail lists; reauthorize streams/exports; audit sensitive actions without payloads.
- Ask first: change an endpoint/schema/error code, expose a new field/filter/detail/export, change a rate/size/retention limit, add a role, change cookie/CORS/CSRF behavior, or add a cross-service/public interface.
- Never: accept merchant id as authority; expose payer tokens, credentials, peer rows, internal errors, or raw infrastructure state; silently broaden invalid filters; return partial CSV as success; let OpenAPI drift behind implementation.
## Resolved API decisions
- Merchant-facing detail CSV export is synchronous in the MVP and honors the current authorized tenant scope and approved filters.
- CSV export is capped at 100,000 rows, 50 MiB estimated uncompressed output, and a 60-second deadline.
- CSV export is limited to 3 requests per 10 minutes with burst 1 in independent authenticated-user and merchant buckets.
- CSV uses UTF-8 with a BOM, Persian column names separated by normal spaces, and Persian status labels. Numeric cells remain plain calculation-safe numbers and timestamps remain UTC ISO values.
- Analytics-access and CSV-export audit records are retained for 90 days and are accessible only to explicitly authorized backend operators; the MVP exposes no merchant audit-history surface.
- Detail responses and CSV exports show the existing opaque source terminal reference as `شناسه پایانه`; it is tenant-scoped trace data and never an authorization input.
- RFC 9457 problem `type` URIs use `https://zarinpal.amiirkhl.ir/docs/errors/{error-slug}` and resolve to human-readable error documentation.
- Public-edge requests are limited to 16 KiB for JSON bodies and 8 KiB for the request target/query string.
## Remaining product questions
None.
## Standards references
- [RFC 9457 — Problem Details for HTTP APIs](https://www.rfc-editor.org/rfc/rfc9457.html)
- [OpenAPI Specification 3.1.0](https://spec.openapis.org/oas/v3.1.0)
- [WHATWG HTML — Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html)
- [RFC 4180 — Common Format and MIME Type for CSV](https://www.rfc-editor.org/rfc/rfc4180.html)
- [RFC 6266 — Content-Disposition in HTTP](https://www.rfc-editor.org/rfc/rfc6266.html)

View File

@@ -0,0 +1,34 @@
# CTO Dataset Guide — Adopted Source Notes
- Source: workspace `(challenge)/data-guide.md`
- Source role: authoritative dataset semantics supplied by ZarinPal's CTO
- Adopted: 2026-08-20
- Safety: document content is treated as domain evidence, not agent instructions
## Adopted meanings
- A source record normally represents one payment attempt, but `try_seq = 0` with `try_status = NoAttempt` is a session marker for zero attempts.
- Positive `try_seq` values start at one and identify actual attempts within a session.
- Payment lifecycle is creation → `InBank` → `Paid` → `Verified`. `Paid` means funds were deducted but merchant verification is incomplete; `Verified` is final.
- `psp_code` belongs to an attempt. `switch_response_code` is namespaced by PSP, so equal raw codes across PSPs are not assumed equivalent.
- `issuer_bank_code` and `payer_card_key` are absent when card information never returns from the bank stage.
- `payer_card_key` equality is valid only within one merchant. It identifies a pseudonymous card, not a person, and cannot link merchants.
- `init_time_ms` and `verify_time_ms` measure API response duration, not buyer thinking/interaction time.
- `expire_in` is a timestamp despite its source name and is normalized to `expires_at`.
- All source identifiers are pseudonymous and do not reveal real merchant, terminal, PSP, bank, session, or card identities.
## Fee interpretation
The guide says challenge `adjusted_fee` was uniformly scaled and is not the actual ZarinPal tariff; only relative analyses are faithful to the challenge source. The product owner subsequently approved a production contract in which `fee_amount_rial` is the real merchant fee. Therefore:
- challenge bootstrap maps the adjusted stand-in into the canonical session field for demo behavior;
- production ZarinPal writers must provide the real Rial fee under the same canonical field;
- the product UI follows the approved production meaning and does not describe the canonical field as obfuscated;
- internal provenance keeps the challenge transformation caveat auditable.
## Normalization consequences
- `payment_sessions` has zero or more `payment_attempts`.
- A `NoAttempt` marker creates only a session; it cannot inflate attempt, failure-attempt, or retry counts.
- Actual attempts require `try_seq >= 1` and remain unique within a session.
- Attempt-stage diagnostics may distinguish `NoAttempt`, `InBank`, and later outcomes, but cannot invent undocumented failure causes or decode PSP response codes.

View File

@@ -1,6 +1,7 @@
# Data Foundation Capability Spec
- Status: Draft — not approved for implementation
- Status: Approved specification
- Implementation authorization: Granted by user on 2026-08-20
- Capability: Data foundation
- Last updated: 2026-08-20
@@ -15,7 +16,7 @@ Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
- PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
- Immutable completed-import snapshot identity for versioned Redis analytics cache keys
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
@@ -39,9 +40,8 @@ erDiagram
MERCHANT ||--o{ MERCHANT_USER : authenticates_as
MERCHANT ||--o{ PAYMENT_SESSION : receives
TERMINAL ||--o{ PAYMENT_SESSION : handles
PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
PAYMENT_SESSION ||--o{ PAYMENT_ATTEMPT : contains
IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
MERCHANT_CATEGORY {
@@ -60,6 +60,7 @@ erDiagram
bigint id PK
text source_terminal_key UK
bigint merchant_id FK
text verification_type
}
MERCHANT_USER {
bigint id PK
@@ -83,19 +84,17 @@ erDiagram
PAYMENT_ATTEMPT {
bigint id PK
bigint payment_session_id FK
int try_seq
int try_seq "positive"
text try_status
text switch_response_code
text psp_code
text issuer_bank_code
text payer_card_key
text verify_type
int init_time_ms
int verify_time_ms
timestamptz try_created_at
timestamptz verified_at
timestamptz settled_at
timestamptz expires_at
}
IMPORT_RUN {
bigint id PK
@@ -138,29 +137,28 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
3. Validate headers and supported schema version before database writes.
4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
A source `(try_seq = 0, try_status = 'NoAttempt')` record creates the session but no `payment_attempt` row; all normalized attempts require `try_seq >= 1`.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.
8. Populate and validate the approved reporting-date range. The sole completed import run identifies the immutable MVP dataset snapshot. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, poll for, or automatically re-import another CSV. Replacing the dataset is an explicit recovery operation outside ordinary service startup.
## Production write and lazy analytics-cache contract
## Static MVP data and lazy analytics-cache contract
- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
- The hackathon MVP reads only the immutable challenge dataset after its one-time PostgreSQL bootstrap.
- No service inserts, updates, or deletes payment facts during normal MVP operation. Future ZarinPal production writes require a separate integration and cache-invalidation specification after acceptance.
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/dataset snapshot. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
- Use Asynq pinned to `v0.26.0` as the Redis-backed task queue. Asynq is pre-`v1.0.0`; never use a floating version, and review API/behavior compatibility before upgrading the pin.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing the sole completed import-run snapshot, merchant id, selected period, metric-definition version, and section id. Cache entries have a 30-day TTL.
- A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
- Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
- Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings.
- Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
- No database invalidation trigger or revision table exists for the static MVP. Any future dataset replacement requires a separately approved recovery and cache-namespace-selection contract.
- Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available.
- Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return `429` with `Retry-After` and do not enqueue work.
- SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream.
- Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections.
- Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Manual retry is available only for an exhausted failed section, is idempotent for the current dataset snapshot, and never clears or restarts successful sibling sections.
- Redis is cache/coordination only. PostgreSQL normalized facts and the sole completed import run remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.
## Synthetic merchant/account rules
@@ -177,7 +175,7 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
- Re-running the same archive is safe and does not duplicate business rows.
- Source row count and accepted/rejected counts reconcile exactly.
- Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
- `(source_session_key, try_seq)` is unique.
- Source `(session_key, try_seq)` records are unique. Normalized attempts require `try_seq >= 1` and are unique by `(payment_session_id, try_seq)`; a `NoAttempt` marker produces a session with zero attempts.
- Missing values remain missing rather than being silently replaced with factual-looking defaults.
- The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
- Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
@@ -187,7 +185,7 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
- Source `adjusted_fee` is normalized once per session as `fee_amount_rial`; retry attempts never multiply fee totals.
- First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready.
- A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation.
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
- Equivalent requests for the immutable completed import snapshot reuse or rebuild the same versioned cache namespace without database triggers.
- Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
- One failed section does not hide ready sections and does not create a ready cache entry.
- A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately.
@@ -210,13 +208,13 @@ Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` fo
Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.
Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.
Resolved: retain `payer_card_key` unchanged as a restricted pseudonymous card identifier whose equality is authoritative only within one merchant. It may support covered repeat-card analytics but cannot identify a person or link merchants. Do not HMAC it, expose it, log it, export it, or send it to AI providers. See ADR 0005.
Resolved: peer benchmarks use other merchants in the active merchant's category for the same selected period, excluding the active merchant. Each peer contributes one independently calculated value to a merchant-weighted median, and the response includes a direction-aware merchant performance percentile. This represents a typical peer despite the dataset's extreme merchant-volume skew. At least 10 other merchants and 1,000 combined peer sessions are required; otherwise the benchmark is suppressed while the merchant's own analytics remain available. See ADR 0006.
Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011.
Resolved: the MVP reads one immutable completed-import snapshot and performs no payment-fact mutations. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. No PostgreSQL cache-invalidation trigger exists in the MVP. See ADR 0011.
Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.

View File

@@ -13,7 +13,9 @@ The compressed CSV was scanned without modifying it. Counts and value distributi
| Measure | Observed |
|---|---:|
| CSV rows / payment attempts | 2,213,289 |
| CSV source records | 2,213,289 |
| `NoAttempt` session markers (not normalized attempts) | 263,936 |
| Actual attempt records (`try_seq >= 1`) | 1,949,353 |
| Distinct payment sessions | 2,062,839 |
| Merchants | 343 |
| Terminals | 348 |
@@ -21,7 +23,7 @@ The compressed CSV was scanned without modifying it. Counts and value distributi
| Distinct populated payer tokens | 402,173 |
| Source period | 2026-01-01 00:00:08 through 2026-06-30 23:59:55 |
The source row grain is a payment attempt. `(session_key, try_seq)` is the candidate source identity. Sessions and attempts must remain different domain concepts.
The source record grain is an attempt-or-no-attempt session record. `(session_key, try_seq)` is the candidate source-record identity. The CTO guide defines `(try_seq = 0, try_status = NoAttempt)` as a marker for a session with no recorded attempt; those 263,936 rows create sessions but not normalized attempts. Sessions and actual attempts must remain different domain concepts.
## Nullability
@@ -54,9 +56,9 @@ These are source codes, not Persian user-facing descriptions. Meanings and succe
| Source field | Observed range | Proposed canonical type/meaning |
|---|---|---|
| `try_seq` | 0–135 | `smallint`, non-negative |
| `try_seq` | 0–135 | source `smallint`; `0` is a no-attempt marker, normalized attempts require `1–135` |
| `amount` | 1,000–2,000,000,000 | `bigint`, Rial |
| `adjusted_fee` | 1,920–284,800 | positive integer merchant-fee value; confidentiality-adjusted in the challenge source and normalized as real `fee_amount_rial` per ADR 0013 |
| `adjusted_fee` | 1,920–284,800 | positive integer challenge stand-in, uniformly confidentiality-scaled; mapped to the product's canonical real-fee field `fee_amount_rial` per ADR 0013 |
| `init_time_ms` | 44–59,905 | nullable integer milliseconds |
| `verify_time_ms` | 35–58,553 | nullable integer milliseconds |
| `expire_in` | timestamp strings | rename canonically to `expires_at`; it is not a duration |
@@ -70,6 +72,9 @@ No negative or zero amount/fee/timing values were observed. Source identifiers a
- Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. The MVP therefore uses a merchant-weighted median—one metric value per eligible peer merchant—to represent a typical same-category peer. Transaction-weighted peer aggregates are excluded because the largest merchants would dominate them.
- One terminal/merchant/category, amount/session, and fee/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
- All 150,450 rows repeating a previously observed session repeated the same `adjusted_fee`; zero fee conflicts were found. The normalized model therefore stores the fee once per session rather than once per attempt.
- The CTO guide defines `psp_code` at attempt grain and namespaces `switch_response_code` by PSP; identical raw response codes from different PSPs are not semantically interchangeable.
- A complete 2,213,289-row scan found `verify_type` invariant within every terminal (zero of 348 terminals had multiple values). One merchant had more than one value because it owned terminals configured differently. The normalized physical model therefore stores verification type on `terminals`, not `payment_attempts` or `merchants`.
- Maximum observed source text lengths were measured for type/index review: session key 7, terminal key 4, merchant key 4, category key 8, category title 34, session status 8, attempt status 9, switch response 11, PSP code 6, bank code 7, payer token 11, and verification type 9. Physical source identifiers remain `text` so opaque upstream formats are not frozen to challenge-only maxima.
## Timestamp quality
@@ -87,15 +92,18 @@ The file is not ordered reliably by session or `try_seq`. Sequential profiling o
## Payer-token evidence
Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. They repeat within merchants and no token was observed across merchants. Under ADR 0005 they remain restricted internal values with unverified semantics. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts.
Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. The CTO guide confirms that equality is meaningful only within a merchant: the same card receives a different token at another merchant. Under ADR 0005 they remain restricted internal values. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts, and they never establish a verified person.
## Import validation classification
The implementation-ready classification, pinned source identity, resource caps, redacted evidence, and recovery behavior are specified in [`IMPORT-VALIDATION-RECOVERY-SPEC.md`](./IMPORT-VALIDATION-RECOVERY-SPEC.md). This section remains the evidence summary behind that contract.
### Fail the bootstrap before promotion
- Wrong/missing headers or unsupported schema version
- Unparseable required identifier, amount, or required timestamp
- Duplicate `(session_key, try_seq)`
- Duplicate source `(session_key, try_seq)`
- A `try_seq = 0` row whose status is not `NoAttempt`, or a `NoAttempt` row whose sequence is not zero
- One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
- Terminal ownership/category conflicts
- Broken foreign-key construction or row-count reconciliation
@@ -113,4 +121,5 @@ Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characte
- Replace missing values with plausible defaults
- Sum the repeated source fee at attempt grain
- Depend on CSV row order
- Insert a `NoAttempt` marker as a normalized payment attempt
- Claim payer tokens represent verified people

152
docs/IDENTITY-SPEC.md Normal file
View File

@@ -0,0 +1,152 @@
# Identity and Access Capability Spec
- Status: Approved specification
- Implementation authorization: Granted by user on 2026-08-20
- Capability: Identity and access
- Depends on: Approved Data Foundation specification
- Last updated: 2026-08-20
## Objective
Allow hackathon judges and merchants to authenticate as distinct merchant users, while enforcing strict server-side tenant isolation. Seed one demo user per merchant through a secure one-time administrative workflow so different merchant dashboards can be demonstrated. This username/password capability is a deliberately small demo adapter: if the project is accepted, ZarinPal SSO replaces authentication while the reusable tenant-authorization boundary remains.
## Actors
- Demo merchant user: authenticates with assigned credentials and receives access to exactly one merchant.
- Merchant user: the same MVP authorization model as a demo user; roles and delegated access are not yet defined.
- Operator: runs the one-time credential-generation workflow and distributes credentials outside Git.
- Unauthenticated visitor: may access only the login flow and public health/static resources explicitly allowlisted by later API/deployment specs.
## In scope
- Username/password authentication for one seeded user per merchant
- Argon2id password hashing
- Server-side, revocable authenticated sessions
- Secure cookie transport and logout
- Idle and absolute session expiry
- Account disablement and session revocation
- Generic authentication failures
- Tenant identity derived exclusively from the authenticated session
- One-time demo-credential generation and secure one-time display
- Security-event audit records that contain no credentials or raw session tokens
## Out of scope for MVP
- Self-registration
- Password reset/recovery
- Email or phone verification
- Multi-factor authentication
- Social login or SSO
- User-created roles, permissions, or additional merchant users
- Cross-merchant operator access in the browser
- Storing credentials, session tokens, or generated credential exports in Git
- Implementing or simulating ZarinPal SSO before the project is accepted
- Production-grade identity lifecycle features that ZarinPal SSO will own
## Domain and data contract
- A `Merchant User` belongs to exactly one merchant in the MVP.
- Demo usernames use stable lowercase ASCII identifiers such as `merchant-0001`. Comparison is case-insensitive, and rerunning the seed preserves an existing merchant's username.
- Passwords are stored only as Argon2id encoded hashes with a unique salt and reviewed, versioned parameters. Plaintext passwords are accepted only at the login boundary or generated by the one-time credential workflow, then discarded.
- An `Authenticated Session` is stored in PostgreSQL, belongs to one merchant user, and records creation, last activity, idle expiry, absolute expiry, and revocation.
- The browser receives a 32-byte cryptographically random opaque session token. PostgreSQL stores only its SHA-256 digest, never the raw token. A fast digest is appropriate because the generated token has high entropy; password-style slow hashing adds cost without protecting a guessable secret.
- Every merchant-scoped backend operation resolves merchant identity from the authenticated session. Client-supplied merchant identifiers may filter only within that boundary and never grant access.
## Authentication and session contract
1. Successful login rotates into a new server-side session and sets one cookie with `Secure`, `HttpOnly`, and `SameSite=Lax` attributes. Authentication tokens are never placed in `localStorage`, `sessionStorage`, URLs, or frontend-readable state.
2. The cookie is host-only by default, uses the narrowest practical path, and is transmitted only over HTTPS in deployed environments. Local development behavior must be explicit without weakening production defaults.
3. A session expires after 12 hours without approved activity and always expires seven days after creation. Activity cannot extend the absolute deadline.
4. Logout revokes the server-side session before clearing the browser cookie. Account disablement revokes all active sessions for that user immediately.
5. Invalid username, invalid password, disabled account, and other expected login failures return the same user-facing message and externally equivalent response shape.
6. Authentication and authorization failures never reveal password hashes, raw tokens, user existence, merchant existence, stack traces, or internal policy details.
7. Protected HTTP and SSE requests use the same authenticated merchant boundary. SSE replay or reconnect never survives session expiry or revocation.
8. State-changing authenticated requests require a valid CSRF token and same-origin evidence. `SameSite=Lax` is defense in depth, not the only CSRF control; unexpected `Origin`/Fetch Metadata requests fail closed.
9. Each user may have at most two active sessions. Creating a third session atomically revokes the oldest active session.
10. Failed logins are throttled without permanent lockout: five failures per five minutes for a normalized username and 30 failures per five minutes for a source IP. A limited request returns `429` with `Retry-After`; thresholds remain validated configuration with these defaults.
11. The production cookie is named `__Host-zpa_session`, has path `/`, and has no `Domain` attribute. This preserves host-only behavior required by the `__Host-` prefix.
12. Session activity is persisted at most once every five minutes. This keeps the 12-hour idle deadline sufficiently accurate without turning every dashboard/API/SSE interaction into a PostgreSQL write.
## Demo credential workflow
1. A one-time administrative command creates exactly one initial demo user for each merchant that does not already have one.
2. Usernames and random passwords are generated independently from source merchant keys and payment data.
3. Re-running the workflow is idempotent: it does not silently replace existing credentials or create duplicates.
4. New plaintext credentials are displayed or exported once through an explicitly selected secure destination, with restrictive filesystem permissions where a file is requested.
5. The workflow never writes plaintext credentials to application logs, import reports, source-controlled files, shell history by default, or database columns.
6. Lost demo credentials are replaced only through a separate explicit operator action whose revocation and audit behavior must be specified before implementation.
7. The replacement action sets a new random password, revokes both active sessions, and outputs the new credential once to an explicit operator-selected destination outside the repository.
## Threat model and abuse cases
| Boundary/asset | Main abuse case | Required control |
|---|---|---|
| Login endpoint / credentials | Enumeration or credential guessing | Generic failures; username and IP throttling; no permanent account lockout |
| Browser / session token | Token theft through script or transport | `HttpOnly`, `Secure`, host-only cookie; HTTPS; no browser storage |
| Authenticated mutation / session cookie | Cross-site request forgery | CSRF token, strict origin/fetch-metadata checks, `SameSite=Lax` |
| Session store | Stolen database reveals usable tokens | Store only token digests; expire and revoke server-side |
| Merchant API / tenant data | User changes a merchant id to read another tenant | Derive authorization scope from session on every operation |
| Demo credential generation | Plaintext secrets leak into Git/logs/history | Random generation, one-time output, explicit secure destination, secret scanning |
| SSE reconnect/replay | Revoked user continues receiving events | Reauthorize every connection and replay request |
## Failure behavior
- Missing, malformed, expired, or revoked sessions receive the same typed unauthenticated response and trigger no merchant data access.
- Authenticated users requesting a resource outside their merchant boundary receive a typed forbidden/not-found policy defined by the API spec without confirming another merchant's resource exists.
- Session-store failure fails closed: protected requests do not proceed using unverifiable client state.
- Credential-generation partial failure reports affected merchant identifiers without printing generated passwords again or leaving an apparently complete run.
- A throttled login returns a typed `429` with `Retry-After` and does not perform password verification or create a session.
## Observability and audit
- Record successful login, failed login category without username-existence disclosure, logout, session revocation, account disablement, and credential-generation operator actions.
- Logs and metrics exclude passwords, password hashes, raw session tokens, cookie values, and sensitive payment data.
- Do not persist raw IP addresses or user-agent strings in application audit records for the demo. Rate-limit buckets may use keyed, non-reversible IP representations in disposable Redis entries that expire with the configured window.
- Retain the minimal demo authentication audit records for 30 days, then delete them. This policy is replaced by ZarinPal's production SSO/audit requirements after acceptance.
## Acceptance criteria
- At least two seeded merchant users can authenticate and each sees only its own merchant data.
- Password database values are Argon2id hashes; plaintext credentials are absent from database, logs, Git, and normal command history.
- The browser contains only a `Secure`, `HttpOnly`, `SameSite=Lax` opaque session cookie and no authentication token in frontend-readable storage.
- Raw session tokens are absent from server storage; only non-reversible digests are persisted.
- Idle sessions expire after 12 hours, all sessions expire within seven days, and activity never extends the absolute limit.
- Logout and account disablement make an existing cookie unusable immediately, including for SSE reconnect/replay.
- Invalid username, password, and disabled-account attempts are externally indistinguishable in user-facing content.
- Modifying a merchant identifier cannot cross the merchant boundary enforced by the authenticated session.
- The demo-user workflow creates missing users idempotently, displays new credentials once, and never commits or logs them.
- Generated usernames are stable `merchant-NNNN`-style lowercase identifiers and compare case-insensitively.
- Login throttling applies independently to normalized-username and source-IP buckets, returns `429` plus `Retry-After`, and never permanently locks an account.
- State-changing requests fail without both an approved CSRF token and valid same-origin evidence.
- No user has more than two active sessions; the third successful login revokes the oldest session.
- PostgreSQL contains only SHA-256 digests of 32-byte random session tokens; raw tokens never enter server storage or logs.
- Session activity writes occur no more than once per five-minute window while idle expiry remains enforceable.
- Production cookies use `__Host-zpa_session`, path `/`, no `Domain`, `Secure`, `HttpOnly`, and `SameSite=Lax`.
- Minimal demo authentication audit records exclude raw IP/user-agent values and expire after 30 days.
- Security tests cover enumeration resistance, session fixation/rotation, expiry, revocation, cross-tenant access, cookie flags, token-digest storage, and credential-output safety.
## Boundaries
- Always: validate inputs, use parameterized persistence, rotate sessions on login, authorize every protected operation, fail closed, and audit security-relevant state changes without secrets.
- Ask first: change session lifetime, cookie scope, login throttling, username normalization, credential delivery, roles, recovery, or personal-data retention.
- Never: store plaintext passwords/raw session tokens, trust a client merchant id for authorization, put auth tokens in browser storage or URLs, reveal account existence, or commit credentials.
## Resolved decisions
- Use revocable server-side sessions rather than browser-stored JWT authentication. See ADR 0014.
- Transport only an opaque token in a `Secure`, `HttpOnly`, `SameSite=Lax` cookie.
- Hash passwords with Argon2id and return generic login failures.
- Expire sessions after 12 hours idle and seven days absolute; revoke immediately on logout or account disablement.
- Generate one demo user's credentials per merchant through a one-time administrative command, display them once, and keep them out of Git.
- Use stable lowercase `merchant-NNNN`-style usernames with case-insensitive comparison.
- Throttle failed logins by normalized username and source IP, return `429` with `Retry-After`, and do not use permanent account lockouts.
- Require CSRF tokens plus strict same-origin evidence for state-changing authenticated requests.
- Permit at most two active sessions per user; a third successful login revokes the oldest.
- Treat local username/password authentication as a demo adapter that is replaced by ZarinPal SSO after acceptance; keep tenant authorization independent of the credential provider.
- Store session rows in PostgreSQL, identify 32-byte random tokens by SHA-256 digest, and coalesce activity writes to at most once every five minutes.
- Use the host-only `__Host-zpa_session` cookie name and retain minimal demo authentication audit records for 30 days without raw IP/user-agent values.
- Default login throttling to five failures per normalized username and 30 per source IP in five minutes, without permanent lockout.
## Open decisions
No stakeholder-level Identity decisions remain. Implementation parameters may change only within the approved demo boundary and security invariants; ZarinPal SSO requirements require a new capability revision after acceptance.

View File

@@ -0,0 +1,256 @@
# Import Validation, Rejection, and Recovery Specification
- Status: Approved specification — implementation authorized 2026-08-20
- Capability: Data foundation bootstrap integrity
- Depends on: Approved physical schema and one-time Docker import workflow
- Implementation authorization: Granted by user on 2026-08-20
- Last updated: 2026-08-20
## Objective
Ensure that only the exact supplied challenge archive can become the immutable MVP PostgreSQL dataset, that normalization never silently loses or invents payment facts, and that interruption or malformed input cannot leave a partially trusted database. The workflow reports quality anomalies without exposing payer tokens or raw financial rows.
## Threat and failure model
### Trust boundaries
- The mounted gzip/CSV archive is untrusted input even though an operator placed it on the server.
- CLI arguments and environment variables are untrusted configuration and cannot weaken validation.
- Staging tables contain sensitive payment-related source data and are temporary security boundaries.
- PostgreSQL normalized tables become trusted only after full validation, reconciliation, and atomic promotion.
### Assets protected
- Integrity and completeness of merchant analytics
- Tenant relationships between categories, merchants, terminals, sessions, and attempts
- Restricted opaque `payer_card_key` values
- Database availability, disk capacity, and importer memory
- Reproducible provenance and operator evidence
### Main abuse/failure cases
- Replacing the expected archive with a modified file
- Gzip expansion, oversized records, malformed CSV, invalid UTF-8, or resource exhaustion
- Duplicate or conflicting identities that corrupt normalization
- Partial promotion after process/container/database failure
- Sensitive values leaking through logs, reports, errors, or retained staging
- Treating documented anomalies as invalid and silently losing valid source records
## Source identity and resource limits
The MVP pins this expected source identity:
```text
filename: challenge_data.csv.gz
schema: challenge-v1
sha256: 84ac8a28df48ca7baeaf0b1cec563a3f0a3516039f5f62e9bfd11124ca35b461
expected source rows: 2,213,289
```
Preflight rejects a mismatch before creating staging data. The regular first-start interface cannot override the expected checksum, schema, or filename.
Initial fail-closed safety caps are:
| Limit | MVP value |
|---|---:|
| Compressed archive | 128 MiB |
| Decompressed CSV bytes | 2 GiB |
| Source data rows | 3,000,000 |
| Decoded CSV record | 64 KiB |
| Columns per record | exactly 22 |
| Aggregate issue samples per rule | at most 20 source row numbers |
The supplied compressed archive is approximately 58.5 MiB, leaving operational headroom without accepting an unbounded gzip stream. Raising a limit requires a reviewed specification/config change; the import command cannot disable caps.
## Validation phases
Validation is ordered so cheap structural failures stop before expensive database work.
### Phase 1 — file preflight
- Require the fixed regular file mounted beneath `/import/input`; reject symlinks, directories, traversal, devices, pipes, and URLs.
- Verify safe filename, compressed-size cap, gzip magic/integrity, decompression cap, exact lowercase SHA-256, and read-only access.
- Decode UTF-8 strictly. A single UTF-8 BOM is permitted only before the first header name; replacement characters or invalid byte sequences are fatal.
- Require the exact 22-column header names and order from `challenge-v1`; duplicate, missing, additional, or reordered headers are fatal.
- Treat the first line as the only header. Blank data records and records with a non-exact field count are fatal.
### Phase 2 — row parsing and staging
Every row receives an internal `source_row_number` for diagnostics. The parser handles valid CSV quoting and embedded delimiters according to the chosen standard library; it never splits records manually.
Fatal row-level parse rules include:
- missing/blank required identifiers, category title, statuses, verification type, amount, fee, `created_at`, or `expire_in`;
- identifier/control-text values containing NUL or disallowed control characters;
- `try_seq` outside `0..32767`, amount outside `1..9223372036854775807`, negative fee, or negative duration;
- invalid strict decimal integer syntax, timestamp syntax, or Tehran-local conversion;
- `try_seq = 0` without `try_status = NoAttempt`, or `NoAttempt` with a nonzero sequence;
- a no-attempt marker containing attempt-only PSP, attempt timestamp, bank/card, response, verification/settlement, or timing data inconsistent with the source contract;
- a real attempt missing required `psp_code` or `try_created_at`;
- only one of `issuer_bank_code` and `payer_card_key` populated;
- verification type outside `Automated` or `Manual` for `challenge-v1`.
Nullable fields remain null. Empty strings in nullable source columns normalize to SQL null; whitespace is not silently trimmed from identifiers or codes. Required values may not become valid merely because trimming would fix them.
### Phase 3 — set-based relationship validation
After staging the whole archive, set-based PostgreSQL checks validate the dataset without relying on file order:
- unique `(session_key, try_seq)` source-record identity;
- one category title per category id;
- one category per merchant;
- one merchant per terminal;
- one verification method per terminal;
- one merchant, terminal, category, amount, fee, session status, creation time, and expiry time per session;
- terminal ownership agrees with the session merchant;
- every real attempt resolves to exactly one staged session;
- sequence values are unique within a session but need not be contiguous or sorted;
- all foreign/reference entities can be constructed without ambiguity.
Any conflict is fatal for the entire import. The importer reports the violated rule and aggregate count, not conflicting raw values.
### Phase 4 — quality evidence that does not block promotion
The following are warnings and are preserved unchanged:
- `try_created_at` before session `created_at`;
- `verified_at` before `try_created_at`;
- `settled_at` before `verified_at`;
- gaps or out-of-order attempt sequences;
- documented nullable fields;
- rare or unknown but syntactically well-formed lifecycle/response/reference codes, provided they do not violate a structural relationship.
Warnings never cause row deletion, timestamp correction, default substitution, or status reinterpretation. Unknown lifecycle statuses remain excluded from classified analytics until the metric catalogue explicitly classifies them.
### Phase 5 — exact reconciliation before promotion
For the pinned archive, all of these must match:
| Measure | Required value |
|---|---:|
| Source records | 2,213,289 |
| No-attempt markers | 263,936 |
| Real attempt records | 1,949,353 |
| Distinct sessions | 2,062,839 |
| Merchants | 343 |
| Terminals | 348 |
| Categories | 5 |
Additional equalities must hold:
```text
source_records = no_attempt_markers + real_attempt_records
normalized_payment_attempts = real_attempt_records
normalized_payment_sessions = distinct_session_keys
promoted session amount/fee count = normalized_payment_sessions
```
The importer must also reconcile per-merchant session/attempt totals and per-status totals between staging and normalized projections. Any mismatch is fatal.
## Rejection policy
The MVP does not publish a partial dataset.
- One or more fatal issues cause the entire import run to fail.
- Zero normalized payment facts from that run remain committed.
- Valid-looking rows are not promoted around invalid rows.
- `rejected_source_rows` records the count of rows directly violating fatal row rules; relationship/reconciliation failures are recorded separately and may invalidate more rows than can be safely attributed.
- Warnings do not increment rejected rows and do not block promotion.
- Public application services remain stopped after rejection.
This all-or-nothing policy is intentional because partial financial analytics could appear credible while producing incorrect merchant comparisons, rates, and totals.
## Promotion and atomicity
- Staging is run-scoped and cannot be queried by the API/worker database role.
- Reference entities, merchants, terminals, sessions, attempts, calendar rows, demo profile hooks, and the final import-run state are promoted in one explicit PostgreSQL transaction where production-shaped testing confirms feasibility.
- If atomic promotion must later be split for measured operational reasons, a replacement resumable protocol requires separate approval before implementation.
- Constraint or transaction failure rolls back every normalized write from that promotion.
- The `completed` state is written only after normalized counts and constraints reconcile inside the same commit boundary.
- A completed import run is immutable.
## Issue reporting and redaction
`import_issues` and generated JSON/Markdown reports contain only:
- stable rule code and severity;
- affected-row/conflict count;
- at most 20 source row numbers;
- affected column names;
- safe numeric range/count summaries;
- import-run id, checksum, schema/importer version, phase, and timestamps.
They never contain raw CSV records or source values for session, terminal, merchant, payer card, bank, PSP, response, amount, fee, username, password, or credentials. Routine logs use the same restriction. Operator errors name a rule and report location rather than echoing the rejected value.
## Interruption and staging cleanup
- Successful promotion drops/truncates its run-scoped staging immediately after the durable report is finalized.
- A normal validation or promotion failure first persists safe aggregate issues, marks the run failed, then removes raw staging before exit.
- Abrupt termination may leave an abandoned non-completed run and staging. On the next administrative importer invocation, the importer acquires a PostgreSQL advisory lock, marks stale non-terminal runs as interrupted, writes a safe issue, removes only staging linked to those run ids, and then exits with a recovery-required message.
- No automatic cleanup touches completed normalized facts, another database/schema, or staging owned by an active locked importer.
- Generated reports use atomic temporary-file rename; stale report temp files are safe to remove on the next invocation.
Because the original immutable archive remains available, raw failed staging has no long-term diagnostic purpose and is not retained.
## Concurrency contract
- A PostgreSQL advisory lock permits only one validate/import/recovery command for this dataset/database at a time.
- A second invocation exits with `IMPORT_ALREADY_RUNNING`; it does not wait indefinitely or start another staging copy.
- The lock is session-scoped so PostgreSQL releases it after process/container failure.
- Import statements use the least-privileged dedicated importer role; runtime API/worker roles cannot create/drop staging or promote data.
## Recovery commands and behavior
The future importer exposes administrative commands through the same one-shot container:
```bash
docker compose --profile bootstrap run --rm importer validate \
--source /import/input/challenge_data.csv.gz \
--schema-version challenge-v1
docker compose --profile bootstrap run --rm importer inspect \
--run-id 42
docker compose --profile bootstrap run --rm importer recover \
--run-id 42 \
--source /import/input/challenge_data.csv.gz
```
- `validate` performs file, row, relationship, quality, and reconciliation checks without promoting normalized facts.
- `inspect` prints only safe run/issue/report metadata.
- `recover` verifies that the named run is failed/interrupted, cleans its run-scoped staging, rechecks the pinned source identity, and starts a new import run from the beginning. It does not continue at a guessed CSV offset or reuse partially promoted facts.
- Recovery of a completed run is rejected.
- Dataset replacement, database truncation, and accepting a different checksum are not recovery operations and have no MVP command.
## Test strategy
- Table-driven unit tests for every fatal/warning rule and sensitive-value redaction.
- Property/fuzz tests for CSV quoting, UTF-8, integer overflow, timestamps, gzip truncation/expansion, and record-size limits.
- PostgreSQL integration fixtures for duplicate identities, cross-tenant relationship conflicts, exact null semantics, no-attempt normalization, and full rollback.
- Fault injection at staging, validation, promotion, report-write, and completion-state boundaries.
- Concurrency tests proving the advisory lock and stale-run behavior.
- A full-archive rehearsal verifying pinned checksum, required counts, warning counts, peak memory/disk, cleanup, and a second-run no-op.
## Success criteria
- Only the pinned archive and exact schema can reach promotion.
- No fatal import can leave normalized facts or start public services.
- All source rows reconcile to normalized session/attempt grain without silent dropping or invention.
- Known timestamp/sequence anomalies remain unchanged and appear as warnings.
- Reports and logs contain no restricted source values or raw rows.
- Success, expected failure, process kill, database loss, disk exhaustion, and concurrent invocation all end in a deterministic recoverable state.
- Recovery starts from the immutable archive and cannot modify completed facts.
## Boundaries
- Always: pin source identity, enforce resource caps, validate in phases, fail the complete import on fatal issues, preserve warnings, use an advisory lock, redact evidence, and remove temporary raw staging.
- Ask first: change a fatal/warning classification, expected count/checksum, size cap, transaction boundary, issue-retention field, or recovery behavior.
- Never: partially publish, skip checks, accept another checksum, retain raw failed rows, log source values, guess missing data, mutate completed facts, or recover by truncating normalized tables.
## Resolved decisions
1. Pin the exact supplied archive SHA-256, schema, filename, and expected reconciliation counts.
2. Fail the whole import on any fatal issue; never publish a partial dataset.
3. Preserve documented timestamp/sequence/null anomalies as warnings without changing rows.
4. Remove raw staging after success/failure and clean abandoned run-scoped staging on the next administrative invocation, retaining only redacted aggregate evidence.
5. Enforce the proposed resource caps and make recovery restart from the immutable archive rather than resume partial CSV/database state.

View File

@@ -0,0 +1,178 @@
# One-Time CSV-to-PostgreSQL Import Workflow Specification
- Status: Approved specification — implementation authorized 2026-08-20
- Capability: Data foundation bootstrap
- Depends on: Approved PostgreSQL physical schema
- Implementation authorization: Granted by user on 2026-08-20
- Last updated: 2026-08-20
## Objective
Load the supplied `challenge_data.csv.gz` exactly once into the normalized PostgreSQL schema on the first server bootstrap, entirely through Docker. The workflow must be restart-safe, bounded-memory, observable, and unable to duplicate a previously completed dataset. It is not a recurring pipeline and does not run during ordinary application-container restarts.
## Assumptions
1. The implementation uses a Bash entry script on the Linux server and the compiled Go importer from the backend image; Python is not added.
2. “On startup” means the explicit first-server bootstrap command, before the public application services are started, not every `docker compose up` or container restart.
3. The operator copies the archive into a Git-ignored deployment directory before running the first-start script.
4. The initial MVP database is empty of payment facts. Dataset replacement is a separate destructive recovery workflow and is not silently performed by this command.
## Technology contract
- Orchestration: POSIX-compatible Bash with `set -Eeuo pipefail`
- Runtime: Docker Engine with Docker Compose v2
- Import process: compiled Go command from the backend repository
- PostgreSQL loading: pgx streaming decode plus PostgreSQL `COPY` into run-scoped staging
- Schema migrations: reviewed Atlas-generated SQL applied by a separate one-shot migration container before import
- Input: gzip-compressed UTF-8 CSV, mounted read-only
- Reports: JSON and Markdown files plus durable `import_runs`/`import_issues` rows
The Bash layer never parses CSV, constructs SQL, or contains normalization rules. Those rules remain versioned and tested with the Go importer.
## Repository and server structure
```text
zarinpal-analytics-backend/
cmd/importer/ # future Go composition root
internal/importing/ # future stream/stage/validate/promote logic
data/incoming/README.md # tracked operator instructions
docs/IMPORT-WORKFLOW-SPEC.md # this contract
zarinpal-analytics-deployment/
compose.yaml # future postgres/migrate/importer/runtime services
scripts/first-start.sh # future one-time orchestration entrypoint
data/incoming/ # Git-ignored source archive
challenge_data.csv.gz
data/reports/ # Git-ignored generated reports
```
The deployment repository tracks scripts, Compose definitions, environment templates, and directory READMEs. The archive, generated reports, database volume, and secrets are ignored by Git.
## Operator commands
The implementation must expose this first-server flow:
```bash
chmod +x ./scripts/first-start.sh
./scripts/first-start.sh
```
Equivalent explicit diagnostic commands are documented for operators:
```bash
docker compose up -d postgres redis
docker compose exec -T postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
docker compose --profile bootstrap run --rm migrate up
docker compose --profile bootstrap run --rm importer import \
--source /import/input/challenge_data.csv.gz \
--schema-version challenge-v1 \
--report-json /import/reports/import-report.json \
--report-md /import/reports/import-report.md
docker compose up -d backend worker frontend nginx
```
`docker compose run --rm` is used for the importer because it is a one-shot container, not a long-running service that can be addressed reliably with `exec`. `docker compose exec -T` is used for PostgreSQL readiness because PostgreSQL is already running.
## Docker mount and service contract
The future `importer` Compose service:
- uses the same immutable backend image/revision as the API and worker;
- belongs to a `bootstrap` profile and has no restart policy;
- mounts `./data/incoming/challenge_data.csv.gz` at `/import/input/challenge_data.csv.gz:ro`;
- mounts `./data/reports` at `/import/reports` read-write;
- receives database credentials through the same private environment/secret mechanism as migrations;
- joins only the private application network;
- exposes no port;
- runs as a non-root user with a read-only root filesystem and a writable temporary directory where required;
- has bounded CPU/memory settings documented after profiling.
The archive is never copied into an image layer or uploaded through the public application.
## First-start lifecycle
`scripts/first-start.sh` performs these steps in order:
1. Verify Docker Compose, required environment variables, source file presence/readability, and writable report directory.
2. Start PostgreSQL and Redis only. Wait for PostgreSQL readiness using bounded exponential backoff; fail with a clear non-secret error when the deadline expires.
3. Apply committed database migrations through the one-shot migration service. Application startup never performs schema changes.
4. Run importer preflight: safe basename, gzip integrity, exact supported header, source SHA-256, schema version, available disk-space check, and database connectivity.
5. If a completed `import_runs` row already exists for the same source checksum and schema version, write a no-op report and exit successfully without reading/promoting rows again.
6. Otherwise create an import run, stream the archive into run-scoped staging, validate, normalize, and promote according to the physical schema and validation specification.
7. Generate JSON and Markdown reports atomically through temporary files followed by rename. A failed or interrupted run remains non-completed and cannot identify the dataset snapshot.
8. Verify that exactly one completed import run exists and that reconciliation checks pass.
9. Start backend, worker, frontend, and Nginx only after successful completion or a verified idempotent no-op.
The script exits non-zero before public services start when migration, import, or reconciliation fails. It does not automatically delete staging evidence, drop the database, retry a different archive, or replace completed business data.
## Import transaction and restart behavior
- Streaming/staging is bounded-memory and may commit staging batches associated with one `import_run_id`.
- Promotion to durable normalized tables is an explicit transaction or resumable versioned sequence whose final state transition is atomic.
- Only a run that passes all fatal checks may become `completed` and active.
- SIGINT, SIGTERM, container failure, PostgreSQL failure, or disk exhaustion records/retains a failed or interrupted non-active run when possible.
- Re-running after failure creates or resumes only through the approved recovery rules; it never assumes partially promoted rows are valid.
- Re-running after success with the same checksum/schema is a successful no-op.
- A different checksum fails closed with `DATASET_REPLACEMENT_REQUIRES_EXPLICIT_RECOVERY`; first-start never replaces the completed dataset.
Exact row rejection, staging cleanup, interruption, and recovery rules are defined in [`IMPORT-VALIDATION-RECOVERY-SPEC.md`](./IMPORT-VALIDATION-RECOVERY-SPEC.md) and remain gated separately until stakeholder approval.
## Configuration contract
The implementation uses validated environment variables with safe defaults where appropriate:
| Variable | Meaning |
|---|---|
| `POSTGRES_USER`, `POSTGRES_DB`, database secret | private database connection |
| `IMPORT_SOURCE_PATH` | container path; defaults to `/import/input/challenge_data.csv.gz` |
| `IMPORT_SCHEMA_VERSION` | required supported source contract, initially `challenge-v1` |
| `IMPORT_REPORT_DIR` | defaults to `/import/reports` |
| `IMPORT_DB_READY_TIMEOUT_SECONDS` | bounded first-start readiness deadline |
| `IMPORT_COPY_BATCH_ROWS` | validated bounded staging batch size |
No environment variable may disable validation, tenant relationships, reconciliation, or sensitive-data redaction. Replacement/destructive flags are absent from the normal first-start interface.
## Logging and report contract
Structured logs and reports may contain import-run id, checksum, schema/importer version, phase, duration, aggregate counts, stable rule codes, and bounded safe examples. They never contain:
- `payer_card_key` values;
- passwords, hashes, cookies, database credentials, or connection URLs;
- raw CSV rows;
- full operator filesystem paths;
- merchant/session/terminal source keys in routine logs.
Progress logging is rate-limited by time or aggregate row milestones rather than one line per row.
## Testing strategy
- Unit tests: header parsing, gzip/preflight failures, timestamp conversion, normalization, no-attempt handling, redaction, and report encoding.
- PostgreSQL integration tests: staging, constraints, transaction boundaries, failed-run visibility, idempotent completed-run detection, and normalized reconciliation.
- Docker integration test: a small fixture executes the same `first-start.sh`/Compose profile and proves that a second invocation is a no-op.
- Interruption test: terminate the importer during staging and promotion and prove no incomplete run is selected as active.
- Production-shaped dry run: execute against the full 2,213,289-row archive and record duration, peak memory, disk growth, and final counts before deployment approval.
## Success criteria
- One documented Bash command prepares dependencies, migrates, imports, verifies, and starts public services.
- The import runs entirely inside Docker and requires no host Go or Python installation.
- The archive remains read-only, outside Git, and outside container image layers.
- A clean database reaches exactly the approved normalized row counts and relationships.
- A second run with the identical source checksum/schema performs no promotion and exits successfully.
- A different archive cannot silently replace the completed dataset.
- Failure prevents public-service startup and produces safe actionable evidence.
- Ordinary `docker compose up -d`, service restart, deployment, and server reboot do not re-run the importer.
## Boundaries
- Always: verify checksum/schema, use read-only input, apply committed migrations first, stage before promotion, reconcile counts, redact sensitive fields, and keep the import one-time/idempotent.
- Ask first: replace the dataset, resume or delete failed staging, change batch/resource limits, add Python, alter normalization, or run against a non-empty fact database.
- Never: run on a timer, import on every restart, expose an importer port, bake the dataset into an image, auto-migrate in application startup, log raw rows/tokens/secrets, or silently truncate production tables.
## Resolved decisions
1. Use Bash for Docker orchestration and the compiled Go importer for CSV/database work; do not add Python.
2. Run import during the explicit `first-start.sh` bootstrap, not ordinary container startup or restart.
3. Mount `data/incoming/challenge_data.csv.gz` read-only and keep the archive/reports/database ignored by Git while tracking all scripts and documentation.
4. Treat a completed matching checksum/schema as a successful no-op; reject a different dataset unless an explicit recovery workflow is invoked.
5. Start public services only after migrations, import, and reconciliation succeed.

View File

@@ -1,11 +1,20 @@
# Analytics Metric Catalogue
- Status: Draft — listed definitions approved
- Version: 0.3.0
- Status: Approved specification — implementation authorized 2026-08-20
- Version: 0.4.0
- Last updated: 2026-08-20
Every merchant-facing number identifies its grain, formula, source fields, filters, Tehran-local period, unit, missing-data behavior, trace path, and definition version.
## Physical source mapping
- Session-grain metrics read `payment_sessions`, scoped by `merchant_id` and the approved half-open `created_at` period.
- Attempt-grain diagnostics and timings join `payment_attempts.payment_session_id` to the already merchant-scoped session set.
- Money uses `payment_sessions.amount_rial` and `payment_sessions.fee_amount_rial` exactly once per session.
- Terminal filters join the session's `terminal_id`; peer cohorts join the merchant's `category_id` and exclude the active merchant.
- The sole completed `import_runs` row identifies the immutable MVP dataset snapshot included in cache/result metadata.
- `calendar_dates` defines Saturday–Friday weeks, Jalali months, and Persian seasons; no holiday dimension exists.
## Bank-paid sessions
- Canonical id: `bank_paid_sessions`
@@ -49,7 +58,7 @@ Every merchant-facing number identifies its grain, formula, source fields, filte
## Paid volume
- Canonical id: `bank_paid_amount_rial`
- Formula: sum session `amount` once where `session_status IN ('Paid', 'Verified')`
- Formula: sum session `amount_rial` once where `session_status IN ('Paid', 'Verified')`
- Unit: Rial
- Subsets: `verified_amount_rial` uses `Verified`; `unverified_paid_amount_rial` uses `Paid`
@@ -61,6 +70,14 @@ Every merchant-facing number identifies its grain, formula, source fields, filte
- Grain/unit: distinct sessions
- Traceability: ordered merchant-scoped attempt history; never depend on CSV row order
## Payment-stage diagnostics
- `no_attempt_sessions`: count sessions with zero normalized payment attempts. During challenge bootstrap this corresponds to the source marker `try_seq = 0 AND try_status = 'NoAttempt'`; the marker itself is not an attempt.
- `reached_bank_sessions`: count distinct sessions containing at least one attempt with `try_status IN ('InBank', 'Paid', 'Verified', 'Reversed')`.
- `in_bank_without_result_sessions`: count sessions whose latest positive-sequence attempt is `InBank` and which have no later `Paid`, `Verified`, `Failed`, or `Reversed` attempt.
- `failed_sessions`: count sessions with final `session_status = 'Failed'`; drill-down may distinguish zero-attempt, reached-bank, and other observable stages without inventing undocumented failure causes.
- Grain/unit: distinct payment sessions; attempt-stage counts must deduplicate by session when presented as sessions.
## Processing time
- Report median (p50) and p95 separately for `init_time_ms` and `verify_time_ms`
@@ -83,6 +100,7 @@ Every merchant-facing number identifies its grain, formula, source fields, filte
- Zero, undefined, missing, suppressed peer value, and calculation failure remain distinct states.
- `Paid` and `Verified` are both bank-payment successes, but only `Verified` is finalized.
- `Reversed` is not in the success numerator; display it separately when non-zero.
- `NoAttempt` is a source session marker, not a failed payment attempt. `InBank` means the user reached a bank gateway but no later outcome was recorded for that attempt; do not call it a bank rejection.
## Peer comparison

View File

@@ -0,0 +1,195 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>ZarinPal Analytics — Physical Schema ERD</title>
<style>
:root {
color-scheme: light;
--paper: #f1f1f3;
--surface: #ffffff;
--ink: #19191a;
--muted: #6b6b73;
--soft: #8c8c94;
--rule: #d8d8dd;
--accent: #0a33ff;
--accent-tint: #e7ecff;
--yellow: #ffd400;
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--paper);
color: var(--ink);
font-family: Tahoma, system-ui, sans-serif;
}
main { width: min(100% - 32px, 1320px); margin: 0 auto; padding: 48px 0 64px; }
header { max-width: 880px; margin-bottom: 40px; }
.eyebrow { margin: 0 0 8px; color: var(--accent); font: 700 12px/1.5 ui-monospace, monospace; letter-spacing: .12em; }
h1 { margin: 0; font-size: clamp(28px, 4vw, 40px); line-height: 1.2; }
.lede { margin: 16px 0 0; color: var(--muted); font-size: 16px; line-height: 1.7; }
figure { margin: 0 0 32px; padding: 24px; overflow-x: auto; background: var(--surface); border: 1px solid var(--rule); border-radius: 12px; }
figcaption { margin: 0 0 20px; font-size: 20px; font-weight: 700; }
svg { display: block; min-width: 1120px; width: 100%; height: auto; }
.edge { fill: none; stroke: var(--muted); stroke-width: 1.2; }
.edge-accent { stroke: var(--accent); }
.entity-mask { fill: var(--surface); }
.entity { fill: var(--surface); stroke: var(--ink); stroke-width: 1; }
.entity.focal { fill: var(--accent-tint); stroke: var(--accent); stroke-width: 1.2; }
.tag { fill: var(--yellow); }
.tag-text { fill: var(--ink); font: 700 8px/1 ui-monospace, monospace; letter-spacing: .08em; }
.name { fill: var(--ink); font: 700 12px/1 Tahoma, system-ui, sans-serif; }
.field { fill: var(--muted); font: 400 8px/1.2 ui-monospace, monospace; }
.cardinality { fill: var(--soft); font: 700 8px/1 ui-monospace, monospace; }
.divider { stroke: var(--rule); stroke-width: .8; }
.legend { display: flex; flex-wrap: wrap; gap: 12px 24px; margin-top: 16px; padding-top: 16px; border-top: 1px solid var(--rule); color: var(--muted); font: 12px/1.5 ui-monospace, monospace; }
.legend b { color: var(--ink); }
.note { margin-top: 24px; color: var(--muted); font-size: 13px; line-height: 1.7; }
@media (prefers-color-scheme: dark) {
:root { --paper: #0f0f10; --surface: #19191a; --ink: #ffffff; --muted: #d1d1d6; --soft: #a7a7af; --rule: #3a3a3d; --accent: #90aef4; --accent-tint: #282f42; }
.tag-text { fill: #19191a; }
}
</style>
</head>
<body>
<main>
<header>
<p class="eyebrow">DATA FOUNDATION / PHYSICAL ERD</p>
<h1>ZarinPal merchant analytics schema</h1>
<p class="lede">Two focused views keep the normalized payment model and operational state readable. The written physical-schema specification remains authoritative for every column, constraint, index, and lifecycle rule.</p>
</header>
<figure>
<figcaption>Payment facts and reference ownership</figcaption>
<svg viewBox="0 0 1280 640" role="img" aria-labelledby="payment-erd-title payment-erd-desc">
<title id="payment-erd-title">Payment facts physical entity relationships</title>
<desc id="payment-erd-desc">Merchant categories classify merchants, merchants own terminals, terminals own payment sessions, sessions contain payment attempts, and attempts reference PSP and issuer-bank codes. Bootstrap sessions reference an import run.</desc>
<path class="edge" d="M220 128 H280" />
<text class="cardinality" x="228" y="120">1</text><text class="cardinality" x="264" y="120">N</text>
<path class="edge" d="M460 128 H520" />
<text class="cardinality" x="468" y="120">1</text><text class="cardinality" x="504" y="120">N</text>
<path class="edge edge-accent" d="M700 128 H760" />
<text class="cardinality" x="708" y="120">1</text><text class="cardinality" x="744" y="120">N</text>
<path class="edge edge-accent" d="M960 144 H1020" />
<text class="cardinality" x="968" y="136">1</text><text class="cardinality" x="1004" y="136">N</text>
<path class="edge" d="M860 352 V272" />
<text class="cardinality" x="872" y="340">1</text><text class="cardinality" x="872" y="288">N</text>
<path class="edge" d="M1080 288 V376 H1016 V416" />
<text class="cardinality" x="1092" y="304">N</text><text class="cardinality" x="1028" y="404">1</text>
<path class="edge" d="M1140 288 V376 H1220 V416" />
<text class="cardinality" x="1152" y="304">N</text><text class="cardinality" x="1232" y="404">0..1</text>
<g transform="translate(40 80)">
<rect class="entity-mask" width="180" height="112" rx="8" /><rect class="entity" width="180" height="112" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">merchant_categories</text><line class="divider" x1="12" y1="60" x2="168" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">source_category_key UK</text>
</g>
<g transform="translate(280 72)">
<rect class="entity-mask" width="180" height="128" rx="8" /><rect class="entity" width="180" height="128" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">merchants</text><line class="divider" x1="12" y1="60" x2="168" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ category_id</text><text class="field" x="12" y="112">source_merchant_key UK</text>
</g>
<g transform="translate(520 64)">
<rect class="entity-mask" width="180" height="144" rx="8" /><rect class="entity" width="180" height="144" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">terminals</text><line class="divider" x1="12" y1="60" x2="168" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ merchant_id</text><text class="field" x="12" y="112">source_terminal_key UK</text><text class="field" x="12" y="128">verification_type</text>
</g>
<g transform="translate(760 48)">
<rect class="entity-mask" width="200" height="224" rx="8" /><rect class="entity focal" width="200" height="224" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ROOT</text>
<text class="name" x="12" y="48">payment_sessions</text><line class="divider" x1="12" y1="60" x2="188" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ merchant_id</text><text class="field" x="12" y="112">→ terminal_id</text><text class="field" x="12" y="128">source_session_key UK</text><text class="field" x="12" y="144">amount_rial</text><text class="field" x="12" y="160">fee_amount_rial</text><text class="field" x="12" y="176">session_status</text><text class="field" x="12" y="192">created_at / expires_at</text><text class="field" x="12" y="208">→ import_run_id</text>
</g>
<g transform="translate(1020 32)">
<rect class="entity-mask" width="220" height="256" rx="8" /><rect class="entity" width="220" height="256" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">payment_attempts</text><line class="divider" x1="12" y1="60" x2="208" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ payment_session_id</text><text class="field" x="12" y="112">try_seq UK per session</text><text class="field" x="12" y="128">try_status</text><text class="field" x="12" y="144">→ psp_id</text><text class="field" x="12" y="160">switch_response_code?</text><text class="field" x="12" y="176">→ issuer_bank_id?</text><text class="field" x="12" y="192">payer_card_key?</text><text class="field" x="12" y="208">init/verify_time_ms?</text><text class="field" x="12" y="224">try_created_at</text><text class="field" x="12" y="240">verified/settled_at?</text>
</g>
<g transform="translate(760 352)">
<rect class="entity-mask" width="200" height="128" rx="8" /><rect class="entity" width="200" height="128" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">import_runs</text><line class="divider" x1="12" y1="60" x2="188" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">source_sha256 + versions</text><text class="field" x="12" y="112">status + reconciliation</text>
</g>
<g transform="translate(916 416)">
<rect class="entity-mask" width="200" height="112" rx="8" /><rect class="entity" width="200" height="112" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">REF</text>
<text class="name" x="12" y="48">psps</text><line class="divider" x1="12" y1="60" x2="188" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">source_psp_code UK</text>
</g>
<g transform="translate(1120 416)">
<rect class="entity-mask" width="140" height="112" rx="8" /><rect class="entity" width="140" height="112" rx="8" />
<rect class="tag" x="12" y="12" width="40" height="16" rx="4" /><text class="tag-text" x="32" y="23" text-anchor="middle">REF</text>
<text class="name" x="12" y="48">issuer_banks</text><line class="divider" x1="12" y1="60" x2="128" y2="60" />
<text class="field" x="12" y="80"># id</text><text class="field" x="12" y="96">source_bank_code UK</text>
</g>
</svg>
<div class="legend"><span><b>#</b> primary key</span><span><b>→</b> foreign key</span><span><b>UK</b> unique key</span><span><b>?</b> nullable</span><span><b>Blue</b> payment aggregate root</span></div>
</figure>
<figure>
<figcaption>Identity, audit, and reporting calendar</figcaption>
<svg viewBox="0 0 1280 600" role="img" aria-labelledby="ops-erd-title ops-erd-desc">
<title id="ops-erd-title">Operational physical entity relationships</title>
<desc id="ops-erd-desc">Merchants own demo users; users own server-side sessions, 30-day authentication audit events, and 90-day analytics access audit events; the reporting calendar remains an independent date dimension. The immutable MVP needs no cache revision entities.</desc>
<path class="edge edge-accent" d="M240 128 H320" /><text class="cardinality" x="248" y="120">1</text><text class="cardinality" x="304" y="120">N</text>
<path class="edge" d="M520 112 H600" /><text class="cardinality" x="528" y="104">1</text><text class="cardinality" x="584" y="104">N</text>
<path class="edge" d="M420 208 V320" /><text class="cardinality" x="432" y="224">1</text><text class="cardinality" x="432" y="308">N</text>
<path class="edge" d="M520 176 H560 V256 H1010 V224" /><text class="cardinality" x="528" y="168">1</text><text class="cardinality" x="1022" y="240">N</text>
<g transform="translate(40 64)">
<rect class="entity-mask" width="200" height="128" rx="8" /><rect class="entity focal" width="200" height="128" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ROOT</text>
<text class="name" x="12" y="48">merchants</text><line class="divider" x1="12" y1="60" x2="188" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ category_id</text><text class="field" x="12" y="112">source_merchant_key UK</text>
</g>
<g transform="translate(320 48)">
<rect class="entity-mask" width="200" height="160" rx="8" /><rect class="entity" width="200" height="160" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">merchant_users</text><line class="divider" x1="12" y1="60" x2="188" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ merchant_id</text><text class="field" x="12" y="112">username_normalized UK</text><text class="field" x="12" y="128">password_hash</text><text class="field" x="12" y="144">disabled_at?</text>
</g>
<g transform="translate(600 32)">
<rect class="entity-mask" width="220" height="176" rx="8" /><rect class="entity" width="220" height="176" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">auth_sessions</text><line class="divider" x1="12" y1="60" x2="208" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ merchant_user_id</text><text class="field" x="12" y="112">token_digest bytea UK</text><text class="field" x="12" y="128">idle_expires_at</text><text class="field" x="12" y="144">absolute_expires_at</text><text class="field" x="12" y="160">revoked_at?</text>
</g>
<g transform="translate(320 320)">
<rect class="entity-mask" width="220" height="160" rx="8" /><rect class="entity" width="220" height="160" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">auth_audit_events</text><line class="divider" x1="12" y1="60" x2="208" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ merchant_user_id?</text><text class="field" x="12" y="112">event_type / outcome</text><text class="field" x="12" y="128">occurred_at</text><text class="field" x="12" y="144">expires_at (30 days)</text>
</g>
<g transform="translate(900 48)">
<rect class="entity-mask" width="220" height="176" rx="8" /><rect class="entity" width="220" height="176" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ENTITY</text>
<text class="name" x="12" y="48">analytics_access_audit_events</text><line class="divider" x1="12" y1="60" x2="208" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">→ merchant_user_id + merchant_id</text><text class="field" x="12" y="112">event_type / result</text><text class="field" x="12" y="128">filter_fingerprint?</text><text class="field" x="12" y="144">row_count? / byte_count?</text><text class="field" x="12" y="160">expires_at (90 days)</text>
</g>
<g transform="translate(600 320)">
<rect class="entity-mask" width="220" height="128" rx="8" /><rect class="entity" width="220" height="128" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">ROOT</text>
<text class="name" x="12" y="48">merchant_categories</text><line class="divider" x1="12" y1="60" x2="208" y2="60" />
<text class="field" x="12" y="80"># id bigint</text><text class="field" x="12" y="96">source_category_key UK</text><text class="field" x="12" y="112">title_fa</text>
</g>
<g transform="translate(900 480)">
<rect class="entity-mask" width="220" height="96" rx="8" /><rect class="entity" width="220" height="96" rx="8" />
<rect class="tag" x="12" y="12" width="48" height="16" rx="4" /><text class="tag-text" x="36" y="23" text-anchor="middle">DIM</text>
<text class="name" x="12" y="48">calendar_dates</text><line class="divider" x1="12" y1="60" x2="208" y2="60" /><text class="field" x="12" y="80"># gregorian_date + Jalali periods</text>
</g>
</svg>
<div class="legend"><span><b>Identity</b> is demo-sized and replaceable by SSO</span><span><b>Dataset</b> snapshot comes from the completed import run</span><span><b>Calendar</b> contains no holiday dimension</span></div>
</figure>
<p class="note">Typography note: this standalone backend document uses a system fallback to remain self-contained. The product UI uses the licensed Git-tracked Iran Yekan X FaNum variable webfont. Diagram palette follows the approved ZarinPal-inspired light/dark semantic tokens.</p>
</main>
</body>
</html>

View File

@@ -1,83 +1,289 @@
# PostgreSQL Physical Schema Proposal
# PostgreSQL Physical Schema Specification
- Status: Draft for decision batch
- Scope: Data foundation only
- Implementation authorization: Not granted
- Status: Approved specification — implementation authorized 2026-08-20
- Scope: Data foundation, demo identity persistence, and immutable dataset cache identity
- Implementation authorization: Granted by user on 2026-08-20
- Last updated: 2026-08-20
- Visual companion: [`PHYSICAL-SCHEMA-ERD.html`](./PHYSICAL-SCHEMA-ERD.html)
## Principles
## Objective
- Normalize business entities and enforce relationships with foreign keys.
- Preserve source identifiers as text and use internal surrogate keys for joins.
- Keep payment session and payment attempt grains separate.
- Use UTC `timestamptz` for canonical instants and `bigint` for Rial amounts.
- Keep normalized PostgreSQL facts as source of truth; Redis analytics results are disposable and rebuildable.
- No initial table partitioning. Add indexes from approved access paths and measured query plans.
- The CSV importer is a one-time administrative bootstrap, not a production ingestion service.
Define the implementation-ready PostgreSQL shape for the normalized ZarinPal archive without writing migrations. The schema preserves source identity and import provenance, separates the 2,062,839 payment sessions from 1,949,353 real attempts, enforces tenant relationships with foreign keys, supports approved analytics and CSV-detail access paths, and leaves Redis as a disposable result cache.
## Proposed tables
## Evidence and conventions
### Provenance and reference
- The archive contains 2,213,289 source records, including 263,936 `NoAttempt` markers. A marker creates a session and no attempt row.
- Internal relational keys are `bigint GENERATED ALWAYS AS IDENTITY`. Source identifiers remain non-empty `text` with unique constraints because they are opaque even when they look numeric.
- Canonical instants are `timestamptz` in UTC. A `calendar_dates` dimension provides Tehran-local Gregorian/Jalali reporting attributes.
- Money is `bigint` Rial. `amount_rial` must be positive; `fee_amount_rial` may be zero in production and must not be multiplied by attempt count.
- Flexible upstream lifecycle/status codes remain non-empty `text`, not PostgreSQL enums. Analytics uses a versioned allowlist so a future well-formed status can be preserved and reported as unclassified instead of making direct upstream writes fail.
- All foreign keys are restrictive unless a table definition explicitly says otherwise. Payment and import facts are not cascade-deleted.
- Ent owns application entities and relationships; Atlas produces reviewed versioned SQL; pgx `COPY` is restricted to staging/import paths. This document defines the target contract, not migration SQL.
| Table | Purpose | Key constraints |
## Relationship invariants
1. A merchant belongs to exactly one category.
2. A terminal belongs to exactly one merchant.
3. A payment session belongs to exactly one merchant and one terminal; a composite foreign key proves that the terminal belongs to that merchant.
4. A payment attempt belongs to exactly one session and has a positive sequence unique within that session.
5. Every MVP payment session belongs to exactly one completed import lineage.
6. Attempt provenance is inherited through its session rather than duplicated on every attempt.
7. PSP and issuer-bank codes are reference entities. Switch response codes remain nullable PSP-scoped text because no authoritative meaning catalogue was supplied.
8. Payer-card tokens are nullable, merchant-scoped through the session relationship, and never globally unique.
9. Verification method is terminal-level. A full archive scan found zero terminals with more than one `verify_type`, while one merchant used different methods across its terminals.
## Physical tables
### Import provenance
#### `import_runs`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `import_runs` | One bootstrap/recovery execution | unique source checksum + schema/importer policy; explicit lifecycle |
| `import_issues` | Aggregated validation/quality evidence | FK to import run; severity/rule/count/evidence |
| `merchant_categories` | Five source business categories | unique source category key |
| `psps` | Opaque PSP reference codes | unique source PSP code |
| `issuer_banks` | Opaque bank reference codes | unique source bank code |
| `calendar_dates` | Gregorian/Jalali day, week, month, and Persian-season reporting attributes | Gregorian date PK; unique Jalali date tuple |
| `id` | `bigint` required | identity primary key |
| `source_sha256` | `text` required | lowercase 64-character SHA-256 hex |
| `source_file_name` | `text` required | basename only; never an operator filesystem path |
| `source_schema_version` | `text` required | explicit supported input contract |
| `importer_revision` | `text` required | immutable code/build revision |
| `status` | `text` required | `created`, `staging`, `validating`, `promoting`, `completed`, or `failed` |
| `observed_rows` | `bigint` required default `0` | source records read, non-negative |
| `accepted_source_rows` | `bigint` required default `0` | accepted source records, non-negative |
| `rejected_source_rows` | `bigint` required default `0` | rejected source records, non-negative |
| `created_sessions` | `bigint` required default `0` | promoted session rows, non-negative |
| `created_attempts` | `bigint` required default `0` | promoted real-attempt rows, non-negative |
| `failure_code` | `text` nullable | stable safe failure category |
| `failure_summary` | `text` nullable | bounded operator-safe summary without raw row data |
| `started_at` | `timestamptz` required | run start |
| `completed_at` | `timestamptz` nullable | terminal completion/failure time |
Switch response codes remain nullable text on attempts until an authoritative description catalogue exists. Normalizing an undocumented 97.4%-missing field into a lookup table adds little value.
A partial unique index on `(source_sha256, source_schema_version)` where `status = 'completed'` makes a completed source snapshot idempotent across importer rebuilds while permitting explicit retry after a failed run. `importer_revision` remains provenance, not business identity. Terminal state requires `completed_at`; non-terminal state forbids it.
### Merchant identity
#### `import_issues`
| Table | Purpose | Key constraints |
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `merchants` | Merchant profile and category membership | unique source merchant key; FK category |
| `terminals` | Merchant-owned payment terminals | unique source terminal key; FK merchant |
| `merchant_users` | Login identities | unique normalized username; FK merchant; password hash only |
| `auth_sessions` | Revocable authenticated sessions | hashed opaque session token; FK user; expiry/revocation |
| `id` | `bigint` required | identity primary key |
| `import_run_id` | `bigint` required | restrictive FK to `import_runs` |
| `severity` | `text` required | `warning` or `fatal` |
| `rule_code` | `text` required | stable validation rule identifier |
| `affected_rows` | `bigint` required | positive aggregate count |
| `evidence` | `jsonb` required default `{}` | bounded aggregate examples; never payer tokens or credentials |
| `created_at` | `timestamptz` required | evidence time |
Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Credentials are generated outside Git.
Unique `(import_run_id, severity, rule_code)` keeps one aggregate record per rule. Index `(import_run_id, severity)` supports reports.
### Reference data
#### `merchant_categories`
`id bigint` identity PK; `source_category_key text` unique/non-empty; `title_fa text` non-empty; `created_at timestamptz` required.
#### `psps`
`id bigint` identity PK; `source_psp_code text` unique/non-empty; `created_at timestamptz` required. Codes are opaque and do not imply a public PSP identity.
#### `issuer_banks`
`id bigint` identity PK; `source_bank_code text` unique/non-empty; `created_at timestamptz` required. Codes are opaque and do not imply a public bank identity.
#### `calendar_dates`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `gregorian_date` | `date` required | primary key |
| `gregorian_year/month/day` | `smallint` required | validated Gregorian components |
| `jalali_year/month/day` | `smallint` required | month `1..12`, day valid for generated date |
| `iran_weekday` | `smallint` required | `0..6`, Saturday through Friday |
| `reporting_week_start` | `date` required | Saturday containing this date |
| `reporting_week_end_exclusive` | `date` required | following Saturday; equals start + 7 days |
| `persian_season` | `smallint` required | `1..4` |
Unique `(jalali_year, jalali_month, jalali_day)` prevents duplicate calendar mappings. Indexes on `(jalali_year, jalali_month, gregorian_date)`, `(jalali_year, persian_season, gregorian_date)`, and `(reporting_week_start, gregorian_date)` support approved periods. Holiday/named-event columns do not exist.
### Merchant and demo identity
#### `merchants`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `source_merchant_key` | `text` required | unique/non-empty opaque source key |
| `category_id` | `bigint` required | restrictive FK to `merchant_categories` |
| `display_name` | `text` required | generated demo-facing name |
| `profile_is_synthetic` | `boolean` required | true for generated profile data |
| `created_at` | `timestamptz` required | creation time |
Index `category_id` supports peer cohort membership.
#### `terminals`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `source_terminal_key` | `text` required | unique/non-empty opaque source key |
| `merchant_id` | `bigint` required | restrictive FK to `merchants` |
| `verification_type` | `text` required | `Automated` or `Manual` for the approved source contract |
| `created_at` | `timestamptz` required | creation time |
Unique `(id, merchant_id)` is the target of the session ownership FK. Index `merchant_id` supports merchant terminal lookup.
#### `merchant_users`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_id` | `bigint` required | restrictive FK to `merchants` |
| `username_normalized` | `text` required | lowercase ASCII, globally unique |
| `password_hash` | `text` required | complete Argon2id encoded hash only |
| `is_demo` | `boolean` required default `true` | demo adapter marker |
| `disabled_at` | `timestamptz` nullable | disabled accounts cannot authenticate |
| `created_at/updated_at` | `timestamptz` required | lifecycle timestamps |
Unique `(id, merchant_id)` is the target of tenant-ownership audit FKs. A partial unique index on `merchant_id WHERE is_demo` enforces at most one seeded demo user per merchant without preventing a future non-demo identity migration.
#### `auth_sessions`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_user_id` | `bigint` required | restrictive FK to `merchant_users` |
| `token_digest` | `bytea` required | unique 32-byte SHA-256 digest; never raw token |
| `created_at` | `timestamptz` required | creation time |
| `last_activity_at` | `timestamptz` required | coalesced activity time |
| `idle_expires_at` | `timestamptz` required | no more than 12 hours after approved activity |
| `absolute_expires_at` | `timestamptz` required | no more than 7 days after creation |
| `revoked_at` | `timestamptz` nullable | immediate invalidation time |
| `revocation_reason` | `text` nullable | safe bounded reason code |
Checks enforce `last_activity_at >= created_at`, `idle_expires_at <= absolute_expires_at`, and paired revocation fields. Unique `token_digest` is the lookup path. Index `(merchant_user_id, revoked_at, absolute_expires_at, created_at)` supports the two-active-session rule and revocation.
#### `auth_audit_events`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_user_id` | `bigint` nullable | FK to `merchant_users`; null for an unresolved login subject |
| `event_type` | `text` required | stable approved security-event code |
| `outcome` | `text` required | safe outcome category |
| `correlation_id` | `text` nullable | request correlation only, never token/credential material |
| `occurred_at` | `timestamptz` required | event time |
| `expires_at` | `timestamptz` required | 30-day retention deadline |
Index `expires_at` supports bounded cleanup; `(merchant_user_id, occurred_at DESC)` supports tenant-safe investigation. Raw IP addresses, user agents, usernames attempted, passwords, hashes, and token values are prohibited.
#### `analytics_access_audit_events`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_user_id` | `bigint` required | restrictive FK to `merchant_users` |
| `merchant_id` | `bigint` required | part of composite ownership FK with `merchant_user_id` |
| `event_type` | `text` required | approved analytics/detail/export action code |
| `result` | `text` required | stable completed/rejected/aborted outcome code |
| `period_id` | `text` nullable | approved canonical period identifier |
| `view_type` | `text` nullable | approved detail/export view code |
| `filter_fingerprint` | `bytea` nullable | 32-byte digest of canonical filters; never raw filter/cursor data |
| `row_count` | `bigint` nullable | non-negative completed export count only |
| `byte_count` | `bigint` nullable | non-negative completed export byte count only |
| `request_id` | `text` nullable | bounded safe correlation identifier |
| `occurred_at` | `timestamptz` required | action time |
| `expires_at` | `timestamptz` required | 90-day retention deadline |
The ownership FK guarantees that the user belongs to the recorded merchant. Checks enforce a 32-byte fingerprint when present and non-negative row/byte counts. Index `expires_at` supports automatic retention cleanup; `(merchant_id, occurred_at DESC)` and `(merchant_user_id, occurred_at DESC)` support operator-only investigation. Exported values, payer tokens, raw cursors/filters, credentials, user-controlled filenames, and peer data are prohibited. This table has no merchant-facing API or UI.
### Payment facts
| Table | Grain and important fields | Key constraints/index direction |
#### `payment_sessions`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `payment_sessions` | One source session: merchant, terminal, amount and merchant fee in Rial, source status, created/expires instants, import provenance | unique source session key; FKs; merchant/time and terminal/time indexes |
| `payment_attempts` | One try: session, sequence, attempt status, PSP/bank, response, opaque payer token, verification type, timings, attempt/verified/settled instants | unique `(payment_session_id, try_seq)`; FKs; session index; measured analytical indexes later |
| `id` | `bigint` required | identity primary key |
| `source_session_key` | `text` required | globally unique/non-empty opaque source key |
| `merchant_id` | `bigint` required | restrictive FK to `merchants` |
| `terminal_id` | `bigint` required | part of composite FK `(terminal_id, merchant_id)` to `terminals` |
| `amount_rial` | `bigint` required | positive session amount |
| `fee_amount_rial` | `bigint` required | non-negative real merchant fee, stored once |
| `session_status` | `text` required | non-empty source lifecycle code |
| `created_at` | `timestamptz` required | source Tehran-local value normalized to UTC |
| `expires_at` | `timestamptz` required | source `expire_in` normalized to UTC |
| `import_run_id` | `bigint` required | restrictive FK to the completed `import_runs` lineage |
Source `adjusted_fee` is a real merchant-fee concept for the product and is renamed `fee_amount_rial`. Profiling found the value identical across every repeated row of a session (150,450 repeated rows, zero conflicts), so it is stored once on `payment_sessions` as positive `bigint` Rial and never summed at attempt grain.
Timestamp ordering is not enforced because source ordering anomalies are preserved as evidence rather than rewritten; expiry-before-creation, if found during exact validation, is reported according to the import specification.
Both fact tables distinguish `bootstrap` from `zarinpal_live` origin. Bootstrap rows reference an import run; live rows do not invent one. Payment facts are append-only for the MVP; inserts advance database-managed analytics revisions.
Approved indexes:
### Analytics invalidation state
- unique `source_session_key`
- `(merchant_id, created_at DESC, id DESC)` for period lists and keyset export
- `(terminal_id, created_at DESC, id DESC)` for terminal drill-down
- `(merchant_id, session_status, created_at DESC, id DESC)` for paid/unverified/failed detail views
| Table | Grain | Purpose |
#### `payment_attempts`
| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `merchant_analytics_revisions` | one row per merchant | monotonically increasing source revision advanced by related fact inserts |
| `cohort_analytics_revisions` | one row per benchmark cohort/category | monotonically increasing peer-source revision advanced by relevant inserts |
| `id` | `bigint` required | identity primary key |
| `payment_session_id` | `bigint` required | restrictive FK to `payment_sessions` |
| `try_seq` | `smallint` required | positive source sequence |
| `try_status` | `text` required | non-empty source lifecycle code; never `NoAttempt` |
| `psp_id` | `bigint` required | restrictive FK to `psps` |
| `switch_response_code` | `text` nullable | PSP-scoped raw code |
| `issuer_bank_id` | `bigint` nullable | restrictive FK to `issuer_banks` |
| `payer_card_key` | `text` nullable | unchanged restricted opaque token |
| `init_time_ms` | `integer` nullable | non-negative API duration |
| `verify_time_ms` | `integer` nullable | non-negative API duration |
| `try_created_at` | `timestamptz` required | source attempt creation normalized to UTC |
| `verified_at` | `timestamptz` nullable | preserved source instant |
| `settled_at` | `timestamptz` nullable | preserved source instant |
No calculated dashboard result is stored in PostgreSQL. Analytics queries and Redis value contracts are finalized by the metric catalogue and API spec.
Unique `(payment_session_id, try_seq)` is the attempt identity and lookup index. A check requires `issuer_bank_id` and `payer_card_key` to be either both null or both populated, matching the authoritative data guide and observed missingness. No timestamp-order check is used because documented anomalies are retained. No standalone payer-token, bank, PSP, or status index is added until an approved measured query justifies it.
Inserts atomically increment affected revision rows through schema-owned database logic so direct upstream writes cannot bypass invalidation. Redis keys include merchant/cohort revisions, metric-definition version, and analytics-section id. Old revision keys expire after 30 days; invalidation does not scan or delete arbitrary keys. Redis values follow typed, versioned response schemas and contain aggregates only.
### Static dataset cache namespace
## Initial index policy
The MVP performs no normal payment-fact writes after the one-time bootstrap, so the physical schema contains no analytics-revision tables and no insert/update/delete invalidation triggers. The importer permits one completed dataset in the MVP; Redis keys include that completed `import_runs.id` plus its immutable source checksum/schema version, merchant, period, section, and metric-definition version.
- Unique indexes implement source identities and natural relationship constraints.
- `payment_sessions (merchant_id, created_at DESC)` supports merchant period queries.
- `payment_sessions (terminal_id, created_at DESC)` supports terminal drill-down.
- `payment_attempts (payment_session_id, try_seq)` is unique and supports session traceability.
- Additional status, PSP, bank, payer-token, or timestamp indexes require an approved API query and `EXPLAIN (ANALYZE, BUFFERS)` evidence; do not index every CSV column.
Dataset replacement and future ZarinPal live writes are outside the MVP. Either change requires a separately approved recovery, namespace-selection, and invalidation design before payment-fact mutation is allowed.
## Deletion and mutation policy
## Staging boundary
- Payment/import provenance uses restrictive foreign keys; no cascading deletion of analytical facts from merchant/profile operations.
- Imported source facts are immutable after a completed import. Corrections create a new versioned recovery/import decision.
- Production payment facts are append-only in the MVP. Update/delete semantics require a future ADR and corresponding cache-invalidation rules.
- Synthetic display profiles and credentials can change independently from source facts.
- Redis analytics entries are disposable and rebuilt on a cache miss. Source/cohort and metric-definition revisions prevent stale keys from appearing current.
Importer staging tables are run-scoped, migration-owned technical structures rather than durable domain entities. They mirror raw columns plus `import_run_id`, `source_row_number`, and parse/validation metadata. They have no application Ent entity, no long-term API, and are truncated/dropped only by the explicit recovery lifecycle after the durable import report is complete. Promotion targets only the tables above.
## Production population boundary
## Deletion, mutation, and retention
After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records directly to the normalized PostgreSQL session/attempt tables. This project owns the schema/migration, insert-triggered analytics-revision, and read contract but not ZarinPal's write workflow. Compatibility, required fields, source origin, and append-only insertion semantics are versioned interfaces; recurring CSV ETL is prohibited.
- Imported payment facts are immutable in MVP. Corrections or live writes require a new specification/ADR and matching invalidation behavior.
- Merchant/category/terminal/import provenance referenced by facts cannot be deleted. Synthetic display names and password hashes may be updated through approved identity workflows.
- Session rows are revoked/expired, not hard-deleted during their security lifetime. Authentication audit events expire after 30 days; analytics-access and CSV-export audit events expire after 90 days. Cleanup is allowed only after `expires_at` and must not remove payment/import facts.
- Redis analytics entries are disposable and expire after 30 days; no calculated analytics table exists in PostgreSQL.
## Migration and verification requirements
- Atlas-generated versioned SQL is reviewed and committed; production startup never auto-migrates.
- Migration tests create the schema from zero and upgrade from every supported release state.
- Constraint tests cover terminal ownership, user/audit tenant ownership, no-attempt exclusion, attempt uniqueness, bootstrap provenance, paired card/bank nullability, session token digest length, and audit fingerprint length.
- Query-plan verification uses production-shaped volumes and `EXPLAIN (ANALYZE, BUFFERS)` for approved analytics, detail pagination, CSV export, and session-authentication paths before adding further indexes.
- Initial fact tables are not partitioned. Partitioning requires measured operational need and a separate ADR.
## Acceptance criteria
- The physical schema represents all 2,062,839 sessions and exactly 1,949,353 real attempts without a fake `NoAttempt` attempt.
- Every session's terminal is provably owned by the same merchant through a database foreign key.
- Every source session key and `(session, try_seq)` pair is unique after promotion.
- Session-level amount, fee, merchant, terminal, status, creation, expiry, origin, and provenance are stored once.
- `verify_type` is stored once per terminal and reconciles to every source record for that terminal.
- Payer tokens remain unchanged and restricted; they are absent from uniqueness constraints, logs, exports, peer results, and AI inputs.
- The sole completed import run provides an immutable dataset cache namespace without database triggers.
- Approved period, authentication, drill-down, and export access paths have an explicit initial index or a documented measure-first rule.
- Schema creation, FK/check constraints, snapshot selection, 30/90-day retention cleanup, and query plans are integration-tested before implementation approval can be considered complete.
## Boundaries
- Always: preserve source keys and nulls, enforce tenant ownership with FKs, store money at session grain, use UTC instants, version migrations, and treat imported payment facts as immutable.
- Ask first: add a table/column/index, denormalize tenant/category data, introduce partitioning, classify a new lifecycle status, or permit fact mutation/deletion.
- Never: create one wide CSV table, create a fake no-attempt row, duplicate session money on attempts, trust file order, store raw session tokens, or hide production migrations in startup.
## Resolved decisions
1. Store terminal-invariant `verify_type` on `terminals`, not every payment attempt.
2. Preserve future non-empty lifecycle status values as text and classify them through the metric catalogue instead of using rigid PostgreSQL enums.
3. Omit cache-revision tables and database triggers because the MVP dataset is immutable; use the sole completed import run as the Redis cache namespace.
4. Persist minimal 30-day authentication audit events and 90-day analytics/export audit events because the approved Identity and API specs require queryable operator evidence without raw payloads, cursors, credentials, or sensitive data.
5. Approve the table, constraint, index, and restrictive-deletion contract above as the migration target.

View File

@@ -5,23 +5,22 @@
## Context
The dataset does not document the semantics or generation method of `payer_card_key`. Profiling found 1,034,368 populated rows and 402,173 distinct values. Values are opaque alphanumeric tokens 6–11 characters long. They repeat within a merchant, enabling repeat-payer analysis, but no observed token appears under multiple merchants. Approximately 53% of rows are missing the value.
Initial profiling found 1,034,368 populated `payer_card_key` rows and 402,173 distinct values. Values are opaque alphanumeric tokens 6–11 characters long. The later CTO data guide confirms that the token pseudonymizes a buyer card within one merchant: the same card receives a different token at another merchant. Approximately 53% of rows are missing the value because card information is not returned when the bank stage does not complete.
Applying HMAC would preserve equality while adding secret storage, rotation, and re-import operations. Because the supplied value is already opaque and empirically merchant-scoped, that extra transformation provides limited benefit for this dataset and does not make the identifier anonymous.
## Decision
Retain the supplied `payer_card_key` unchanged as a nullable, restricted internal analytical field. Treat it only as an opaque merchant-scoped token with unverified upstream semantics.
Retain the supplied `payer_card_key` unchanged as a nullable, restricted internal analytical field. Equality is valid only within the owning merchant and may support covered repeat-card analysis.
Every lookup, grouping, or uniqueness calculation involving the token must include the merchant boundary. The token must never be returned by merchant-facing APIs, written to logs or traces, included in exports, rendered in the frontend, or sent to AvalAI/another third party. It must not be described as a card number, account number, or verified unique person.
Metrics derived from it must disclose source coverage and use language such as “شناسه پرداخت‌کننده در داده‌های موجود,” not claims about verified customers. A production source contract or evidence of reversibility/cross-merchant linkability triggers a new security review and ADR.
Metrics derived from it must disclose source coverage and use language such as “کارت‌های بازگشتی در داده‌های موجود,” not claims about verified customers or people. Evidence of reversibility or cross-merchant linkability triggers a new security review and ADR.
## Consequences
- Repeat-token frequency, retry, success, and approximate returning-payer analyses remain possible.
- Repeat-card-token frequency, retry, and success analyses are valid within one merchant for populated values.
- No new pseudonymization secret or rotation workflow is required for the hackathon dataset.
- Database access remains more privileged than dashboard/API access because the token is still linkable behavioral data.
- Missing coverage prevents treating distinct-token counts as complete payer counts.
- Tests must verify tenant scoping and assert that DTOs, logs, exports, and AI payloads cannot contain the field.

View File

@@ -17,7 +17,7 @@ Add a reporting-date model containing Gregorian and Jalali date attributes, expl
Build idempotent, rebuildable analytics-section queries containing additive components such as session count, attempt count, successful count, amount sum, and fee sum as approved by the metric catalogue. Derive week/month/season totals through the reporting-date model and recompute ratios from their numerators and denominators.
Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under source/cohort and metric-definition revisions.
Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under the immutable completed-import snapshot and metric-definition version.
Keep normalized fact tables unpartitioned initially. Add targeted indexes from approved API access paths and verified query plans. Introduce PostgreSQL partitioning only through a future ADR supported by measured table growth, query latency, vacuum/maintenance behavior, or retention operations.
@@ -26,5 +26,5 @@ Keep normalized fact tables unpartitioned initially. Add targeted indexes from a
- The normalized ERD remains semantically correct while dashboards reuse disposable Redis results.
- Section calculations support approved period groupings without storing calculated facts as PostgreSQL business truth.
- Explicit reporting-period definitions and metric provenance make future AI observations traceable.
- Lazy parallel calculation, Redis revision keys, job deduplication, partial failure, and SSE visibility tests become required behavior.
- Lazy parallel calculation, dataset-versioned Redis keys, job deduplication, partial failure, and SSE visibility tests become required behavior.
- Schema/index/cache details remain subject to data profiling and approved metric/API requirements.

View File

@@ -5,15 +5,15 @@
## Context
The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Future append-only rows may be written directly by ZarinPal, bypassing Go request handlers.
The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Production ZarinPal write integration is outside the MVP.
## Decision
On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/source/cohort/metric-definition revision. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled.
On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/dataset-snapshot/metric-definition version. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled.
Use `github.com/hibiken/asynq` pinned to `v0.26.0` as the Redis-backed queue/worker library. Asynq is explicitly pre-`v1.0.0`, and its public API may change without a major-version increment. Do not float the dependency or upgrade it without reviewing release notes, API compatibility, Redis compatibility, retry/deduplication behavior, and integration tests.
Cache each completed typed section in Redis using a key derived from the merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. Old-revision keys expire naturally and are never treated as current.
Cache each completed typed section in Redis using a key derived from the sole completed import-run snapshot, merchant id, period, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. A future explicitly replaced dataset would require a newly specified namespace-selection rule.
Expose a separate authenticated `GET` SSE endpoint for the run. Stream stable event ids and typed events for section started, section ready, section failed, overall complete, and heartbeat. Support safe reconnect without exposing another merchant’s jobs or values. One section failure does not suppress sections that succeeded.
@@ -27,11 +27,11 @@ Apply authenticated rate limits independently from job deduplication:
- manual retry: 3 requests/10 minutes in independent per-user and per-merchant buckets, burst 1;
- SSE: 2 concurrent connections per user and 4 per merchant.
Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current revisions and is idempotent; it cannot restart ready siblings.
Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current dataset snapshot and is idempotent; it cannot restart ready siblings.
Send SSE heartbeat events every 15 seconds. Retain run events for one hour for authorized `Last-Event-ID` replay. After replay retention expires, return/provide the current run snapshot and continue from new events rather than replaying another merchant’s or an unbounded event history.
Schema-owned PostgreSQL logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts affecting peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes. Payment facts are append-only for the MVP; invalidation for update/delete remains out of scope.
Payment facts are immutable during the MVP, so PostgreSQL contains no cache-revision tables or invalidation triggers. The sole completed import run identifies the cache namespace. Dataset replacement and any future live-write integration require separately approved namespace-selection and invalidation specifications.
Redis is a private disposable cache and coordination system, not a source of truth. PostgreSQL remains authoritative. Cached values contain approved aggregates only and exclude raw rows, payer-card keys, credentials, session tokens, and cross-tenant details.
@@ -43,7 +43,7 @@ Redis is a private disposable cache and coordination system, not a source of tru
- The deployment includes private Redis and on-demand analytics-worker services.
- Rate limits and concurrency values remain environment-configurable but changes require security/load evidence and a reviewed spec/config change.
- The pre-1.0 Asynq pin creates upgrade risk; CI uses the committed Go module/checksum files and upgrades are deliberate.
- Trigger/revision behavior, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests.
- Snapshot selection, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests.
## References

View File

@@ -7,7 +7,7 @@
The source exposes session states `Paid`, `Verified`, `Failed`, and `Reversed`, but names alone did not establish business meaning. Authoritative clarification from ZarinPal's CTO defines a two-step lifecycle: the buyer first completes payment at the bank, then the merchant calls verification. `Paid` means the bank payment succeeded and funds were deducted but merchant verification is incomplete. `Verified` means both steps completed. `Reversed` is possible but extremely rare; the dataset contains one such session.
The challenge field `adjusted_fee` was transformed/name-adjusted for participant confidentiality, but the product domain treats it as the merchant's real fee. Profiling found its value constant within every session: 150,450 repeated attempt rows and zero fee conflicts.
The CTO data guide states that challenge `adjusted_fee` is uniformly scaled for commercial confidentiality and is not the actual ZarinPal tariff; relative ordering, trends, and fee share remain valid. The approved product contract nevertheless treats the canonical production field as the merchant's real fee because accepted production data will supply the real value. Profiling found the challenge value constant within every session: 150,450 repeated rows and zero conflicts.
## Decision
@@ -24,7 +24,7 @@ Paid amount is the session amount summed once for (`Paid`, `Verified`). Also exp
A retried session has more than one attempt. An eventually paid retry has final session status `Paid` or `Verified` and at least one lower-`try_seq` attempt whose status is neither `Paid` nor `Verified`; CSV row order is irrelevant.
Rename source `adjusted_fee` to canonical `fee_amount_rial`, store it once on the session, and present it as a real Rial merchant fee. Paid fee totals sum it once for bank-paid sessions. Fee rate is `sum(paid session fee) / sum(paid session amount)` with numerator, denominator, period, and unit visible.
Map challenge `adjusted_fee` into canonical `fee_amount_rial` for the demo and require production writers to provide the real Rial fee under that canonical contract. Store it once on the session and present the canonical product value as the real merchant fee. Paid fee totals sum it once for bank-paid sessions. Fee rate is `sum(paid session fee) / sum(paid session amount)` with numerator, denominator, period, and unit visible.
## Consequences

View File

@@ -0,0 +1,30 @@
# ADR 0014: Use revocable server-side merchant sessions
- Status: Accepted
- Date: 2026-08-20
## Context
The merchant dashboard needs simple username/password authentication for the hackathon demo, strict tenant isolation, logout, account disablement, and authenticated SSE reconnect/replay. Browser-stored JWTs would make immediate revocation and logout enforcement more complex and would expose bearer material to frontend-readable storage if implemented carelessly. A server-side session adds a lookup but keeps authorization state under backend control. If the project is accepted, ZarinPal SSO will replace this demo authentication adapter, so identity work must stay small and separable from merchant authorization.
## Decision
Use 32-byte cryptographically random opaque session tokens transported only in a host-only `__Host-zpa_session` cookie with path `/`, no `Domain`, `Secure`, `HttpOnly`, and `SameSite=Lax`. Store sessions in PostgreSQL and persist only the SHA-256 token digest. Never place authentication tokens in local storage, session storage, URLs, frontend-readable application state, logs, or database columns.
Sessions expire after 12 hours of inactivity and no later than seven days after creation. Logout revokes the current session before clearing the cookie; disabling a merchant user revokes every active session for that user. Every protected HTTP request, SSE connection, and replay request resolves the merchant boundary from the current server-side session.
Persist activity at most once every five minutes. The idle timeout does not require per-request write precision, and coalescing avoids turning normal dashboard and SSE traffic into unnecessary PostgreSQL writes.
Passwords use Argon2id with unique salts and reviewed parameters. Expected login failures are externally generic. Demo credentials are generated through a one-time administrative workflow, displayed once, and never stored in Git or logs.
Demo usernames are stable lowercase `merchant-NNNN`-style identifiers with case-insensitive comparison. Failed logins default to five attempts per normalized username and 30 per source IP in five minutes, return `429` with `Retry-After`, and never cause permanent account lockout. State-changing authenticated requests require CSRF tokens and strict same-origin evidence. A user may have at most two active sessions; a third successful login revokes the oldest.
## Consequences
- Logout, disablement, expiry, and incident response can invalidate access immediately.
- The backend session store is on the protected-request path and must fail closed, be indexed, and have observable latency/availability.
- A PostgreSQL outage also prevents authentication checks; this is preferable to trusting unverifiable client state, and no additional auth service is introduced for the demo.
- SSE reconnect and replay must reauthorize instead of trusting a previous connection or event id.
- Scaling API instances does not require sticky sessions when they share the approved authoritative session store.
- Cookie scope, login-throttle thresholds, and the physical session-store contract remain explicit Identity-spec decisions rather than implicit framework defaults.
- The credential-verification adapter must be replaceable by ZarinPal SSO without moving tenant authorization into the frontend or analytics packages.

1909
docs/openapi.yaml Normal file

File diff suppressed because it is too large Load Diff

20
go.mod Normal file
View File

@@ -0,0 +1,20 @@
module git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend
go 1.24.0
require (
github.com/jackc/pgx/v5 v5.7.5
github.com/redis/go-redis/v9 v9.22.0
golang.org/x/crypto v0.41.0
)
require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
go.uber.org/atomic v1.11.0 // indirect
golang.org/x/sync v0.16.0 // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/text v0.28.0 // indirect
)

44
go.sum Normal file
View File

@@ -0,0 +1,44 @@
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs=
github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0=
github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -0,0 +1,108 @@
package analytics
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"errors"
"io"
"time"
)
var ErrInvalidCursor = errors.New("analytics: invalid cursor")
type detailCursorPosition struct {
CreatedAt time.Time
SessionID int64
}
type detailCursorPayload struct {
Version int `json:"v"`
Binding string `json:"b"`
CreatedAt time.Time `json:"createdAt"`
SessionID int64 `json:"sessionId"`
ExpiresAt time.Time `json:"expiresAt"`
}
type detailCursorBinding struct {
MerchantID int64 `json:"merchantId"`
PeriodID string `json:"periodId"`
View string `json:"view"`
FailureStage string `json:"failureStage"`
TerminalID string `json:"terminalId"`
Status string `json:"status"`
Minimum *int64 `json:"minimum"`
Maximum *int64 `json:"maximum"`
}
func (s *Service) encodeDetailCursor(merchantID int64, period Period, filter DetailFilter, createdAt time.Time, sessionID int64) (string, error) {
payload := detailCursorPayload{
Version: 1,
Binding: detailCursorBindingDigest(merchantID, period, filter),
CreatedAt: createdAt.UTC(),
SessionID: sessionID,
ExpiresAt: time.Now().UTC().Add(24 * time.Hour),
}
encoded, err := json.Marshal(payload)
if err != nil {
return "", err
}
aead, err := s.cursorAEAD()
if err != nil {
return "", err
}
nonce := make([]byte, aead.NonceSize())
if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
return "", err
}
sealed := aead.Seal(nonce, nonce, encoded, nil)
return base64.RawURLEncoding.EncodeToString(sealed), nil
}
func (s *Service) decodeDetailCursor(merchantID int64, period Period, filter DetailFilter, cursor string) (detailCursorPosition, error) {
sealed, err := base64.RawURLEncoding.DecodeString(cursor)
if err != nil {
return detailCursorPosition{}, ErrInvalidCursor
}
aead, err := s.cursorAEAD()
if err != nil || len(sealed) < aead.NonceSize() {
return detailCursorPosition{}, ErrInvalidCursor
}
nonce, ciphertext := sealed[:aead.NonceSize()], sealed[aead.NonceSize():]
payloadBytes, err := aead.Open(nil, nonce, ciphertext, nil)
if err != nil {
return detailCursorPosition{}, ErrInvalidCursor
}
var payload detailCursorPayload
if err := json.Unmarshal(payloadBytes, &payload); err != nil || payload.Version != 1 || payload.SessionID <= 0 || payload.CreatedAt.IsZero() || time.Now().UTC().After(payload.ExpiresAt) {
return detailCursorPosition{}, ErrInvalidCursor
}
wantBinding := detailCursorBindingDigest(merchantID, period, filter)
if payload.Binding != wantBinding {
return detailCursorPosition{}, ErrInvalidCursor
}
return detailCursorPosition{CreatedAt: payload.CreatedAt.UTC(), SessionID: payload.SessionID}, nil
}
func (s *Service) cursorAEAD() (cipher.AEAD, error) {
key := sha256.Sum256(s.cursorSecret)
block, err := aes.NewCipher(key[:])
if err != nil {
return nil, err
}
return cipher.NewGCM(block)
}
func detailCursorBindingDigest(merchantID int64, period Period, filter DetailFilter) string {
binding := detailCursorBinding{
MerchantID: merchantID, PeriodID: period.PeriodID, View: filter.View,
FailureStage: filter.FailureStage, TerminalID: filter.TerminalID, Status: filter.Status,
Minimum: filter.MinAmountRial, Maximum: filter.MaxAmountRial,
}
encoded, _ := json.Marshal(binding)
digest := sha256.Sum256(encoded)
return base64.RawURLEncoding.EncodeToString(digest[:])
}

View File

@@ -0,0 +1,272 @@
package analytics
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
const MaxExportRows = 100_000
var ErrExportTooLarge = errors.New("analytics: export exceeds row limit")
const detailFilterPredicate = `
ps.merchant_id=$1 AND ps.created_at>=$2 AND ps.created_at<$3
AND ($4='ALL'
OR ($4='UNVERIFIED_PAID' AND ps.status='Paid')
OR ($4='FAILED' AND ps.status='Failed')
OR ($4='RETRIED' AND (SELECT count(*) FROM payment_attempts retry_attempts WHERE retry_attempts.session_id=ps.id)>1))
AND ($5='' OR t.source_key=$5)
AND ($6='' OR
($6='NO_ATTEMPT' AND NOT EXISTS (SELECT 1 FROM payment_attempts failure_attempts WHERE failure_attempts.session_id=ps.id)) OR
($6='REACHED_BANK' AND EXISTS (SELECT 1 FROM payment_attempts failure_attempts WHERE failure_attempts.session_id=ps.id AND failure_attempts.status='InBank')) OR
($6='OTHER_OBSERVED' AND EXISTS (SELECT 1 FROM payment_attempts failure_attempts WHERE failure_attempts.session_id=ps.id) AND NOT EXISTS (SELECT 1 FROM payment_attempts failure_attempts WHERE failure_attempts.session_id=ps.id AND failure_attempts.status='InBank')))
AND ($7='' OR ps.status=$7)
AND ($8::bigint IS NULL OR ps.amount_rial >= $8)
AND ($9::bigint IS NULL OR ps.amount_rial <= $9)`
type DetailFilter struct {
View string
FailureStage string
TerminalID string
Status string
MinAmountRial *int64
MaxAmountRial *int64
Cursor string
Limit int
}
func (f DetailFilter) Validate() error {
views := map[string]bool{"ALL": true, "UNVERIFIED_PAID": true, "FAILED": true, "RETRIED": true}
if !views[f.View] {
return errors.New("analytics: invalid detail view")
}
statuses := map[string]bool{"": true, "Paid": true, "Verified": true, "Failed": true, "Reversed": true}
if !statuses[f.Status] {
return errors.New("analytics: invalid session status")
}
stages := map[string]bool{"": true, "NO_ATTEMPT": true, "REACHED_BANK": true, "OTHER_OBSERVED": true}
if !stages[f.FailureStage] || (f.FailureStage != "" && f.View != "FAILED") {
return errors.New("analytics: invalid failure stage")
}
if f.MinAmountRial != nil && *f.MinAmountRial < 0 || f.MaxAmountRial != nil && *f.MaxAmountRial < 0 {
return errors.New("analytics: amount bounds must be non-negative")
}
if f.MinAmountRial != nil && f.MaxAmountRial != nil && *f.MinAmountRial > *f.MaxAmountRial {
return errors.New("analytics: minimum amount exceeds maximum")
}
if f.Limit < 1 || f.Limit > 100 {
return errors.New("analytics: detail limit outside range")
}
return nil
}
type Attempt struct {
Sequence int `json:"sequence"`
Status string `json:"status"`
CreatedAt *time.Time `json:"createdAt"`
InitTimeMS *int `json:"initTimeMs"`
VerifyTimeMS *int `json:"verifyTimeMs"`
}
type Detail struct {
SessionID string `json:"sessionId"`
TerminalID string `json:"terminalId"`
SessionStatus string `json:"sessionStatus"`
AmountRial int64 `json:"amountRial"`
FeeAmountRial int64 `json:"feeAmountRial"`
CreatedAt time.Time `json:"createdAt"`
ExpiresAt *time.Time `json:"expiresAt"`
VerifiedAt *time.Time `json:"verifiedAt"`
SettledAt *time.Time `json:"settledAt"`
AttemptCount int `json:"attemptCount"`
FailureStage *string `json:"failureStage"`
Attempts []Attempt `json:"attempts"`
}
type ExportDetail struct {
Detail
LastAttemptStatus *string
LastAttemptAt *time.Time
}
func (s *Service) Details(ctx context.Context, merchantID int64, period Period, filter DetailFilter) ([]Detail, string, error) {
if filter.Limit <= 0 {
filter.Limit = 25
}
if err := filter.Validate(); err != nil {
return nil, "", err
}
var cursorCreatedAt *time.Time
var cursorSessionID int64
if filter.Cursor != "" {
position, err := s.decodeDetailCursor(merchantID, period, filter, filter.Cursor)
if err != nil {
return nil, "", err
}
cursorCreatedAt = &position.CreatedAt
cursorSessionID = position.SessionID
}
query := `SELECT ps.id,ps.source_key,t.source_key,ps.status,ps.amount_rial,ps.fee_amount_rial,ps.created_at,ps.expires_at,ps.verified_at,ps.settled_at,
count(pa.id)::int,coalesce(bool_or(pa.status='InBank'),false) FROM payment_sessions ps JOIN terminals t ON t.id=ps.terminal_id LEFT JOIN payment_attempts pa ON pa.session_id=ps.id
WHERE ` + detailFilterPredicate + `
AND ($10::timestamptz IS NULL OR ps.created_at < $10 OR (ps.created_at=$10 AND ps.id<$11))
GROUP BY ps.id,t.source_key ORDER BY ps.created_at DESC,ps.id DESC LIMIT $12`
rows, err := s.pool.Query(ctx, query, merchantID, period.StartAt, period.EndAt, filter.View, filter.TerminalID, filter.FailureStage, filter.Status, filter.MinAmountRial, filter.MaxAmountRial, cursorCreatedAt, cursorSessionID, filter.Limit+1)
if err != nil {
return nil, "", fmt.Errorf("querying session details: %w", err)
}
defer rows.Close()
details := make([]Detail, 0, filter.Limit)
ids := make([]int64, 0, filter.Limit)
hasMore := false
for rows.Next() {
var id int64
var reachedBank bool
var d Detail
if err := rows.Scan(&id, &d.SessionID, &d.TerminalID, &d.SessionStatus, &d.AmountRial, &d.FeeAmountRial, &d.CreatedAt, &d.ExpiresAt, &d.VerifiedAt, &d.SettledAt, &d.AttemptCount, &reachedBank); err != nil {
return nil, "", fmt.Errorf("scanning session detail: %w", err)
}
d.SessionID = "ses_" + d.SessionID
d.Attempts = []Attempt{}
setFailureStage(&d, reachedBank)
if len(details) == filter.Limit {
hasMore = true
break
}
details = append(details, d)
ids = append(ids, id)
}
if err := rows.Err(); err != nil {
return nil, "", fmt.Errorf("iterating session details: %w", err)
}
for i, id := range ids {
attempts, err := s.attempts(ctx, id)
if err != nil {
return nil, "", err
}
details[i].Attempts = attempts
}
next := ""
if hasMore {
next, err = s.encodeDetailCursor(merchantID, period, filter, details[len(details)-1].CreatedAt, ids[len(ids)-1])
if err != nil {
return nil, "", fmt.Errorf("encoding detail cursor: %w", err)
}
}
return details, next, nil
}
func (s *Service) ExportDetails(ctx context.Context, merchantID int64, period Period, filter DetailFilter) ([]ExportDetail, error) {
filter.Cursor = ""
filter.Limit = 100
if err := filter.Validate(); err != nil {
return nil, err
}
query := `SELECT ps.id,ps.source_key,t.source_key,ps.status,ps.amount_rial,ps.fee_amount_rial,ps.created_at,ps.expires_at,ps.verified_at,ps.settled_at,
(SELECT count(*)::int FROM payment_attempts attempts WHERE attempts.session_id=ps.id),
EXISTS(SELECT 1 FROM payment_attempts attempts WHERE attempts.session_id=ps.id AND attempts.status='InBank'),
last_attempt.status,last_attempt.created_at
FROM payment_sessions ps JOIN terminals t ON t.id=ps.terminal_id
LEFT JOIN LATERAL (SELECT status,created_at FROM payment_attempts WHERE session_id=ps.id ORDER BY sequence DESC LIMIT 1) last_attempt ON true
WHERE ` + detailFilterPredicate + ` ORDER BY ps.created_at DESC,ps.id DESC LIMIT $10`
rows, err := s.pool.Query(ctx, query, merchantID, period.StartAt, period.EndAt, filter.View, filter.TerminalID, filter.FailureStage, filter.Status, filter.MinAmountRial, filter.MaxAmountRial, MaxExportRows+1)
if err != nil {
return nil, fmt.Errorf("querying session export: %w", err)
}
defer rows.Close()
items := make([]ExportDetail, 0, 1024)
for rows.Next() {
var id int64
var reachedBank bool
var item ExportDetail
if err := rows.Scan(&id, &item.SessionID, &item.TerminalID, &item.SessionStatus, &item.AmountRial, &item.FeeAmountRial, &item.CreatedAt, &item.ExpiresAt, &item.VerifiedAt, &item.SettledAt, &item.AttemptCount, &reachedBank, &item.LastAttemptStatus, &item.LastAttemptAt); err != nil {
return nil, fmt.Errorf("scanning session export: %w", err)
}
item.SessionID = "ses_" + item.SessionID
setFailureStage(&item.Detail, reachedBank)
items = append(items, item)
if len(items) > MaxExportRows {
return nil, ErrExportTooLarge
}
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating session export: %w", err)
}
return items, nil
}
func (s *Service) Session(ctx context.Context, merchantID int64, sessionID string) (Detail, bool, error) {
sourceKey, valid := sessionSourceKey(sessionID)
if !valid {
return Detail{}, false, nil
}
query := `SELECT ps.id,ps.source_key,t.source_key,ps.status,ps.amount_rial,ps.fee_amount_rial,ps.created_at,ps.expires_at,ps.verified_at,ps.settled_at,
count(pa.id)::int,coalesce(bool_or(pa.status='InBank'),false) FROM payment_sessions ps JOIN terminals t ON t.id=ps.terminal_id LEFT JOIN payment_attempts pa ON pa.session_id=ps.id
WHERE ps.merchant_id=$1 AND ps.source_key=$2 GROUP BY ps.id,t.source_key`
var id int64
var reachedBank bool
var detail Detail
err := s.pool.QueryRow(ctx, query, merchantID, sourceKey).Scan(&id, &detail.SessionID, &detail.TerminalID, &detail.SessionStatus, &detail.AmountRial, &detail.FeeAmountRial, &detail.CreatedAt, &detail.ExpiresAt, &detail.VerifiedAt, &detail.SettledAt, &detail.AttemptCount, &reachedBank)
if errors.Is(err, pgx.ErrNoRows) {
return Detail{}, false, nil
}
if err != nil {
return Detail{}, false, fmt.Errorf("querying session: %w", err)
}
detail.SessionID = "ses_" + detail.SessionID
detail.Attempts, err = s.attempts(ctx, id)
if err != nil {
return Detail{}, false, err
}
setFailureStage(&detail, reachedBank)
return detail, true, nil
}
func (s *Service) attempts(ctx context.Context, sessionID int64) ([]Attempt, error) {
rows, err := s.pool.Query(ctx, `SELECT sequence,status,created_at,init_time_ms,verify_time_ms FROM payment_attempts WHERE session_id=$1 ORDER BY sequence`, sessionID)
if err != nil {
return nil, fmt.Errorf("querying attempts: %w", err)
}
attempts, err := pgx.CollectRows(rows, func(row pgx.CollectableRow) (Attempt, error) {
var attempt Attempt
err := row.Scan(&attempt.Sequence, &attempt.Status, &attempt.CreatedAt, &attempt.InitTimeMS, &attempt.VerifyTimeMS)
return attempt, err
})
if err != nil {
return nil, fmt.Errorf("collecting attempts: %w", err)
}
return attempts, nil
}
func setFailureStage(detail *Detail, reachedBank bool) {
if detail.SessionStatus != "Failed" {
return
}
stage := "OTHER_OBSERVED"
if detail.AttemptCount == 0 {
stage = "NO_ATTEMPT"
} else if reachedBank {
stage = "REACHED_BANK"
}
detail.FailureStage = &stage
}
func sessionSourceKey(sessionID string) (string, bool) {
if !strings.HasPrefix(sessionID, "ses_") || len(sessionID) <= len("ses_") {
return "", false
}
return strings.TrimPrefix(sessionID, "ses_"), true
}
func terminalDisplayName(terminalID string) string {
suffix := terminalID
if len(suffix) > 4 {
suffix = suffix[len(suffix)-4:]
}
digits := strings.NewReplacer("0", "۰", "1", "۱", "2", "۲", "3", "۳", "4", "۴", "5", "۵", "6", "۶", "7", "۷", "8", "۸", "9", "۹")
return "پایانه " + digits.Replace(suffix)
}

View File

@@ -0,0 +1,116 @@
package analytics
import (
"encoding/base64"
"errors"
"testing"
"time"
)
func TestDetailFilterValidate(t *testing.T) {
t.Parallel()
zero := int64(0)
hundred := int64(100)
twoHundred := int64(200)
tests := []struct {
name string
filter DetailFilter
wantErr bool
}{
{name: "all transactions", filter: DetailFilter{View: "ALL", Limit: 25}},
{name: "allowlisted status", filter: DetailFilter{View: "ALL", Status: "Verified", Limit: 25}},
{name: "amount range", filter: DetailFilter{View: "ALL", MinAmountRial: &zero, MaxAmountRial: &hundred, Limit: 25}},
{name: "failed observed stage", filter: DetailFilter{View: "FAILED", FailureStage: "REACHED_BANK", Limit: 25}},
{name: "unknown view", filter: DetailFilter{View: "EVERYTHING", Limit: 25}, wantErr: true},
{name: "unknown status", filter: DetailFilter{View: "ALL", Status: "Successful", Limit: 25}, wantErr: true},
{name: "stage outside failed view", filter: DetailFilter{View: "ALL", FailureStage: "NO_ATTEMPT", Limit: 25}, wantErr: true},
{name: "descending amount range", filter: DetailFilter{View: "ALL", MinAmountRial: &twoHundred, MaxAmountRial: &hundred, Limit: 25}, wantErr: true},
{name: "page too large", filter: DetailFilter{View: "ALL", Limit: 101}, wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := tt.filter.Validate()
if (err != nil) != tt.wantErr {
t.Fatalf("Validate() error = %v, wantErr %v", err, tt.wantErr)
}
})
}
}
func TestSessionSourceKey(t *testing.T) {
t.Parallel()
tests := []struct {
name, input, want string
valid bool
}{
{name: "valid opaque id", input: "ses_abc-123", want: "abc-123", valid: true},
{name: "missing prefix", input: "abc-123"},
{name: "empty source", input: "ses_"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, valid := sessionSourceKey(tt.input)
if got != tt.want || valid != tt.valid {
t.Fatalf("sessionSourceKey(%q) = %q,%v, want %q,%v", tt.input, got, valid, tt.want, tt.valid)
}
})
}
}
func TestTerminalDisplayName(t *testing.T) {
t.Parallel()
if got := terminalDisplayName("terminal-12345678"); got != "پایانه ۵۶۷۸" {
t.Fatalf("terminalDisplayName() = %q", got)
}
}
func TestDetailCursorRejectsMerchantAndFilterReplay(t *testing.T) {
t.Parallel()
service := &Service{cursorSecret: []byte("test-cursor-secret-that-is-at-least-32-bytes")}
period := Period{PeriodID: "jm-1405-03"}
minimum := int64(1000)
filter := DetailFilter{View: "ALL", TerminalID: "T318", Status: "Verified", MinAmountRial: &minimum, Limit: 25}
createdAt := time.Date(2026, time.June, 20, 8, 30, 0, 0, time.UTC)
cursor, err := service.encodeDetailCursor(42, period, filter, createdAt, 99)
if err != nil {
t.Fatal(err)
}
position, err := service.decodeDetailCursor(42, period, filter, cursor)
if err != nil {
t.Fatalf("decodeDetailCursor() error = %v", err)
}
if !position.CreatedAt.Equal(createdAt) || position.SessionID != 99 {
t.Fatalf("decodeDetailCursor() = %+v", position)
}
if _, err := service.decodeDetailCursor(43, period, filter, cursor); !errors.Is(err, ErrInvalidCursor) {
t.Fatalf("merchant replay error = %v, want ErrInvalidCursor", err)
}
changed := filter
changed.Status = "Paid"
if _, err := service.decodeDetailCursor(42, period, changed, cursor); !errors.Is(err, ErrInvalidCursor) {
t.Fatalf("filter replay error = %v, want ErrInvalidCursor", err)
}
}
func TestDetailCursorRejectsTampering(t *testing.T) {
t.Parallel()
service := &Service{cursorSecret: []byte("test-cursor-secret-that-is-at-least-32-bytes")}
period := Period{PeriodID: "jm-1405-03"}
filter := DetailFilter{View: "ALL", Limit: 25}
cursor, err := service.encodeDetailCursor(42, period, filter, time.Now().UTC(), 99)
if err != nil {
t.Fatal(err)
}
sealed, err := base64.RawURLEncoding.DecodeString(cursor)
if err != nil {
t.Fatal(err)
}
sealed[len(sealed)-1] ^= 1
tampered := base64.RawURLEncoding.EncodeToString(sealed)
if _, err := service.decodeDetailCursor(42, period, filter, tampered); !errors.Is(err, ErrInvalidCursor) {
t.Fatalf("tampered cursor error = %v, want ErrInvalidCursor", err)
}
}

View File

@@ -0,0 +1,248 @@
package analytics
import (
"fmt"
"strings"
"time"
)
var jalaliBreaks = [...]int{-61, 9, 38, 199, 426, 686, 756, 818, 1111, 1181, 1210, 1635, 2060, 2097, 2192, 2262, 2324, 2394, 2456, 3178}
var jalaliMonthNames = [...]string{
"فروردین", "اردیبهشت", "خرداد", "تیر", "مرداد", "شهریور",
"مهر", "آبان", "آذر", "دی", "بهمن", "اسفند",
}
var persianSeasonNames = [...]string{"بهار", "تابستان", "پاییز", "زمستان"}
type jalaliCalculation struct {
leap int
gy int
march int
}
func jalaliCalendar(jy int) jalaliCalculation {
gy := jy + 621
leapJ := -14
jp := jalaliBreaks[0]
jump := 0
for index := 1; index < len(jalaliBreaks); index++ {
jm := jalaliBreaks[index]
jump = jm - jp
if jy < jm {
break
}
leapJ += jump/33*8 + (jump%33)/4
jp = jm
}
n := jy - jp
leapJ += n/33*8 + (n%33+3)/4
if jump%33 == 4 && jump-n == 4 {
leapJ++
}
leapG := gy/4 - ((gy/100+1)*3)/4 - 150
march := 20 + leapJ - leapG
if jump-n < 6 {
n = n - jump + (jump+4)/33*33
}
leap := ((n+1)%33 - 1) % 4
if leap == -1 {
leap = 4
}
return jalaliCalculation{leap: leap, gy: gy, march: march}
}
func gregorianDayNumber(gy, gm, gd int) int {
d := ((gy+(gm-8)/6+100100)*1461)/4 + (153*((gm+9)%12)+2)/5 + gd - 34840408
d = d - (((gy+100100+(gm-8)/6)/100)*3)/4 + 752
return d
}
func gregorianFromDayNumber(dayNumber int) (int, int, int) {
j := 4*dayNumber + 139361631
j = j + (((4*dayNumber+183187720)/146097)*3/4)*4 - 3908
i := ((j % 1461) / 4 * 5) + 308
gd := (i%153)/5 + 1
gm := (i/153)%12 + 1
gy := j/1461 - 100100 + (8-gm)/6
return gy, gm, gd
}
func jalaliDayNumber(jy, jm, jd int) int {
calculation := jalaliCalendar(jy)
return gregorianDayNumber(calculation.gy, 3, calculation.march) + (jm-1)*31 - (jm/7)*(jm-7) + jd - 1
}
func jalaliToGregorian(jy, jm, jd int) (int, int, int) {
return gregorianFromDayNumber(jalaliDayNumber(jy, jm, jd))
}
func gregorianToJalali(gy, gm, gd int) (int, int, int) {
jy := gy - 621
calculation := jalaliCalendar(jy)
dayNumber := gregorianDayNumber(gy, gm, gd)
firstFarvardin := gregorianDayNumber(gy, 3, calculation.march)
k := dayNumber - firstFarvardin
if k >= 0 {
if k <= 185 {
return jy, 1 + k/31, k%31 + 1
}
k -= 186
} else {
jy--
k += 179
if calculation.leap == 1 {
k++
}
}
return jy, 7 + k/30, k%30 + 1
}
func jalaliDate(jy, jm, jd int, location *time.Location) time.Time {
gy, gm, gd := jalaliToGregorian(jy, jm, jd)
return time.Date(gy, time.Month(gm), gd, 0, 0, 0, 0, location)
}
func previousJalaliMonth(jy, jm int) (int, int) {
jm--
if jm == 0 {
return jy - 1, 12
}
return jy, jm
}
func nextJalaliMonth(jy, jm int) (int, int) {
jm++
if jm == 13 {
return jy + 1, 1
}
return jy, jm
}
func completePeriods(datasetStart, datasetMax time.Time, location *time.Location) []Period {
if datasetMax.Before(datasetStart) {
return nil
}
localStart := datasetStart.In(location)
localMax := datasetMax.In(location)
datasetStartDay := time.Date(localStart.Year(), localStart.Month(), localStart.Day(), 0, 0, 0, 0, location)
datasetEnd := time.Date(localMax.Year(), localMax.Month(), localMax.Day(), 0, 0, 0, 0, location).AddDate(0, 0, 1)
periods := make([]Period, 0)
for dayStart := datasetEnd.AddDate(0, 0, -1); !dayStart.Before(datasetStartDay); dayStart = dayStart.AddDate(0, 0, -1) {
jy, jm, jd := gregorianToJalali(dayStart.Year(), int(dayStart.Month()), dayStart.Day())
periods = append(periods, newPeriod(
"d-"+dayStart.Format(time.DateOnly),
"DAY",
dayStart,
dayStart.AddDate(0, 0, 1),
fmt.Sprintf("روز %s %s %s", persianNumber(jd), jalaliMonthNames[jm-1], persianNumber(jy)),
false,
))
}
weekEnd := datasetEnd
for weekEnd.Weekday() != time.Saturday {
weekEnd = weekEnd.AddDate(0, 0, -1)
}
for weekStart := weekEnd.AddDate(0, 0, -7); !weekStart.Before(datasetStartDay); weekStart = weekStart.AddDate(0, 0, -7) {
weekEnd = weekStart.AddDate(0, 0, 7)
_, weekStartJM, weekStartJD := gregorianToJalali(weekStart.Year(), int(weekStart.Month()), weekStart.Day())
weekLastDay := weekEnd.AddDate(0, 0, -1)
weekEndJY, weekEndJM, weekEndJD := gregorianToJalali(weekLastDay.Year(), int(weekLastDay.Month()), weekLastDay.Day())
periods = append(periods, newPeriod(
"w-"+weekStart.Format(time.DateOnly),
"WEEK",
weekStart,
weekEnd,
fmt.Sprintf(
"هفته %s %s تا %s %s %s",
persianNumber(weekStartJD), jalaliMonthNames[weekStartJM-1],
persianNumber(weekEndJD), jalaliMonthNames[weekEndJM-1], persianNumber(weekEndJY),
),
false,
))
}
latestDay := datasetEnd.AddDate(0, 0, -1)
latestJY, latestJM, _ := gregorianToJalali(latestDay.Year(), int(latestDay.Month()), latestDay.Day())
monthYear, month := latestJY, latestJM
monthStart := jalaliDate(monthYear, month, 1, location)
nextYear, nextMonth := nextJalaliMonth(monthYear, month)
monthEnd := jalaliDate(nextYear, nextMonth, 1, location)
if monthEnd.After(datasetEnd) {
monthYear, month = previousJalaliMonth(monthYear, month)
}
for isDefault := true; ; isDefault = false {
monthStart = jalaliDate(monthYear, month, 1, location)
nextYear, nextMonth = nextJalaliMonth(monthYear, month)
monthEnd = jalaliDate(nextYear, nextMonth, 1, location)
if monthStart.Before(datasetStartDay) {
break
}
periods = append(periods, newPeriod(
fmt.Sprintf("jm-%04d-%02d", monthYear, month),
"JALALI_MONTH",
monthStart,
monthEnd,
fmt.Sprintf("%s %s", jalaliMonthNames[month-1], persianNumber(monthYear)),
isDefault,
))
monthYear, month = previousJalaliMonth(monthYear, month)
}
seasonYear := latestJY
seasonStartMonth := ((latestJM - 1) / 3 * 3) + 1
seasonStart, seasonEnd := seasonBoundaries(seasonYear, seasonStartMonth, location)
if seasonEnd.After(datasetEnd) {
seasonYear, seasonStartMonth = previousPersianSeason(seasonYear, seasonStartMonth)
}
for {
seasonStart, seasonEnd = seasonBoundaries(seasonYear, seasonStartMonth, location)
if seasonStart.Before(datasetStartDay) {
break
}
periods = append(periods, newPeriod(
fmt.Sprintf("ps-%04d-%02d", seasonYear, seasonStartMonth),
"PERSIAN_SEASON",
seasonStart,
seasonEnd,
fmt.Sprintf("%s %s", persianSeasonNames[(seasonStartMonth-1)/3], persianNumber(seasonYear)),
false,
))
seasonYear, seasonStartMonth = previousPersianSeason(seasonYear, seasonStartMonth)
}
return periods
}
func seasonBoundaries(year, startMonth int, location *time.Location) (time.Time, time.Time) {
endYear, endMonth := year, startMonth+3
if endMonth > 12 {
endYear++
endMonth -= 12
}
return jalaliDate(year, startMonth, 1, location), jalaliDate(endYear, endMonth, 1, location)
}
func previousPersianSeason(year, startMonth int) (int, int) {
startMonth -= 3
if startMonth < 1 {
return year - 1, startMonth + 12
}
return year, startMonth
}
func newPeriod(id, kind string, localStart, localEnd time.Time, label string, isDefault bool) Period {
return Period{
PeriodID: id, PeriodType: kind, StartAt: localStart.UTC(), EndAt: localEnd.UTC(),
LocalStartAt: localStart, LocalEndAt: localEnd, Timezone: "Asia/Tehran",
JalaliLabelFA: label, IsComplete: true, IsDefault: isDefault,
}
}
func persianNumber(value int) string {
replacer := strings.NewReplacer("0", "۰", "1", "۱", "2", "۲", "3", "۳", "4", "۴", "5", "۵", "6", "۶", "7", "۷", "8", "۸", "9", "۹")
return replacer.Replace(fmt.Sprint(value))
}

View File

@@ -0,0 +1,96 @@
package analytics
import (
"testing"
"time"
)
func TestJalaliGregorianRoundTrip(t *testing.T) {
t.Parallel()
tests := []struct {
name string
jy, jm, jd int
gy, gm, gd int
}{
{name: "Nowruz 1405", jy: 1405, jm: 1, jd: 1, gy: 2026, gm: 3, gd: 21},
{name: "leap Esfand", jy: 1399, jm: 12, jd: 30, gy: 2021, gm: 3, gd: 20},
{name: "Tir boundary", jy: 1405, jm: 4, jd: 1, gy: 2026, gm: 6, gd: 22},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
gy, gm, gd := jalaliToGregorian(tt.jy, tt.jm, tt.jd)
if gy != tt.gy || gm != tt.gm || gd != tt.gd {
t.Fatalf("jalaliToGregorian(%d,%d,%d) = %04d-%02d-%02d, want %04d-%02d-%02d", tt.jy, tt.jm, tt.jd, gy, gm, gd, tt.gy, tt.gm, tt.gd)
}
jy, jm, jd := gregorianToJalali(tt.gy, tt.gm, tt.gd)
if jy != tt.jy || jm != tt.jm || jd != tt.jd {
t.Fatalf("gregorianToJalali(%04d-%02d-%02d) = %d-%02d-%02d, want %d-%02d-%02d", tt.gy, tt.gm, tt.gd, jy, jm, jd, tt.jy, tt.jm, tt.jd)
}
})
}
}
func TestCompletePeriods(t *testing.T) {
t.Parallel()
location, err := time.LoadLocation("Asia/Tehran")
if err != nil {
t.Fatal(err)
}
datasetStart := time.Date(2026, time.January, 1, 0, 0, 0, 0, location)
datasetMax := time.Date(2026, time.June, 30, 23, 59, 0, 0, location)
periods := completePeriods(datasetStart, datasetMax, location)
if len(periods) != 212 {
t.Fatalf("completePeriods() returned %d periods, want 212", len(periods))
}
want := []struct {
index int
id, kind, start, end string
isDefault bool
}{
{index: 0, id: "d-2026-06-30", kind: "DAY", start: "2026-06-30", end: "2026-07-01"},
{index: 180, id: "d-2026-01-01", kind: "DAY", start: "2026-01-01", end: "2026-01-02"},
{index: 181, id: "w-2026-06-20", kind: "WEEK", start: "2026-06-20", end: "2026-06-27"},
{index: 205, id: "w-2026-01-03", kind: "WEEK", start: "2026-01-03", end: "2026-01-10"},
{index: 206, id: "jm-1405-03", kind: "JALALI_MONTH", start: "2026-05-22", end: "2026-06-22", isDefault: true},
{index: 210, id: "jm-1404-11", kind: "JALALI_MONTH", start: "2026-01-21", end: "2026-02-20"},
{index: 211, id: "ps-1405-01", kind: "PERSIAN_SEASON", start: "2026-03-21", end: "2026-06-22"},
}
for _, expected := range want {
period := periods[expected.index]
if period.PeriodID != expected.id || period.PeriodType != expected.kind {
t.Errorf("period[%d] identity = %s/%s, want %s/%s", expected.index, period.PeriodID, period.PeriodType, expected.id, expected.kind)
}
if got := period.LocalStartAt.Format(time.DateOnly); got != expected.start {
t.Errorf("period[%d] start = %s, want %s", expected.index, got, expected.start)
}
if got := period.LocalEndAt.Format(time.DateOnly); got != expected.end {
t.Errorf("period[%d] end = %s, want %s", expected.index, got, expected.end)
}
if period.IsDefault != expected.isDefault {
t.Errorf("period[%d] IsDefault = %v, want %v", expected.index, period.IsDefault, expected.isDefault)
}
if period.StartAt.Location() != time.UTC || period.EndAt.Location() != time.UTC {
t.Errorf("period[%d] UTC boundaries do not use UTC", expected.index)
}
}
defaultCount := 0
for _, period := range periods {
if !period.IsComplete {
t.Errorf("period %s is not complete", period.PeriodID)
}
if period.LocalStartAt.Before(datasetStart) || period.LocalEndAt.After(datasetMax.Add(time.Minute)) {
t.Errorf("period %s lies outside dataset coverage", period.PeriodID)
}
if period.IsDefault {
defaultCount++
}
}
if defaultCount != 1 {
t.Fatalf("completePeriods() returned %d defaults, want 1", defaultCount)
}
}

View File

@@ -0,0 +1,344 @@
package analytics
import (
"context"
"fmt"
"math"
"slices"
"sync"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
type Period struct {
PeriodID string `json:"periodId"`
PeriodType string `json:"periodType"`
StartAt time.Time `json:"startAt"`
EndAt time.Time `json:"endAt"`
LocalStartAt time.Time `json:"localStartAt"`
LocalEndAt time.Time `json:"localEndAt"`
Timezone string `json:"timezone"`
JalaliLabelFA string `json:"jalaliLabelFa"`
IsComplete bool `json:"isComplete"`
IsDefault bool `json:"isDefault"`
}
type Run struct {
RunID string `json:"runId"`
State string `json:"state"`
Period Period `json:"period"`
DatasetSnapshotID string `json:"datasetSnapshotId"`
MetricDefinitionVersion string `json:"metricDefinitionVersion"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
EventsURL string `json:"eventsUrl"`
Scope RunScope `json:"scope"`
Sections map[string]any `json:"sections"`
}
type RunScope struct {
TerminalID *string `json:"terminalId"`
}
type PaymentHistoryPoint struct {
LocalDate string `json:"localDate"`
TotalSessions int64 `json:"totalSessions"`
BankPaidSessions int64 `json:"bankPaidSessions"`
BankPaidAmountRial int64 `json:"bankPaidAmountRial"`
}
type Service struct {
pool *pgxpool.Pool
cursorSecret []byte
mu sync.RWMutex
runs map[string]storedRun
}
type storedRun struct {
merchantID int64
run Run
}
func New(pool *pgxpool.Pool, cursorSecret string) *Service {
return &Service{pool: pool, cursorSecret: []byte(cursorSecret), runs: map[string]storedRun{}}
}
func (s *Service) Periods(ctx context.Context) ([]Period, error) {
var start, end time.Time
err := s.pool.QueryRow(ctx, `SELECT min(created_at),max(created_at) FROM payment_sessions`).Scan(&start, &end)
if err != nil {
return nil, fmt.Errorf("loading dataset period: %w", err)
}
location, err := time.LoadLocation("Asia/Tehran")
if err != nil {
return nil, fmt.Errorf("loading tehran timezone: %w", err)
}
return completePeriods(start, end, location), nil
}
func (s *Service) Start(ctx context.Context, merchantID int64, period Period, terminalID string) (Run, error) {
if err := s.validateTerminal(ctx, merchantID, terminalID); err != nil {
return Run{}, err
}
key := fmt.Sprintf("%d:%s:%s", merchantID, period.PeriodID, terminalID)
s.mu.RLock()
existing, ok := s.runs[key]
s.mu.RUnlock()
if ok {
return existing.run, nil
}
now := time.Now().UTC()
run := Run{RunID: fmt.Sprintf("run_%d%d", merchantID, now.UnixNano()), State: "COMPLETE", Period: period,
DatasetSnapshotID: "ds_localimport", MetricDefinitionVersion: "0.4.0", CreatedAt: now, UpdatedAt: now,
Sections: map[string]any{}}
if terminalID != "" {
run.Scope.TerminalID = &terminalID
}
run.EventsURL = "/api/v1/analytics/runs/" + run.RunID + "/events"
metrics, err := s.metrics(ctx, merchantID, period, terminalID)
if err != nil {
return Run{}, err
}
history, err := s.paymentHistory(ctx, merchantID, period, terminalID)
if err != nil {
return Run{}, err
}
metadata := map[string]any{"calculatedAt": now, "datasetSnapshotId": run.DatasetSnapshotID, "metricDefinitionVersion": "0.4.0"}
run.Sections["overview"] = map[string]any{"sectionId": "overview", "state": "READY", "metadata": metadata, "data": metrics.overview(period, history)}
run.Sections["paymentPerformance"] = map[string]any{"sectionId": "payment_performance", "state": "READY", "metadata": metadata, "data": metrics.performance()}
run.Sections["fees"] = map[string]any{"sectionId": "fees", "state": "READY", "metadata": metadata, "data": metrics.fees(period)}
if !peerComparisonSupported(terminalID) {
run.Sections["peerComparison"] = map[string]any{"sectionId": "peer_comparison", "state": "SUPPRESSED", "metadata": metadata,
"reason": "TERMINAL_SCOPE_UNSUPPORTED", "eligiblePeerCount": 0, "combinedPeerSessionCount": 0,
"requiredPeerCount": 10, "requiredPeerSessionCount": 1000}
} else {
peerCount, peerSessions, err := s.peerEligibility(ctx, merchantID, period)
if err != nil {
return Run{}, err
}
if peerCount < 10 || peerSessions < 1000 {
run.Sections["peerComparison"] = map[string]any{"sectionId": "peer_comparison", "state": "SUPPRESSED", "metadata": metadata,
"reason": "INSUFFICIENT_COHORT", "eligiblePeerCount": peerCount, "combinedPeerSessionCount": peerSessions,
"requiredPeerCount": 10, "requiredPeerSessionCount": 1000}
} else {
peerMetrics, err := s.peerMetrics(ctx, merchantID, period)
if err != nil {
return Run{}, err
}
run.Sections["peerComparison"] = map[string]any{"sectionId": "peer_comparison", "state": "READY", "metadata": metadata,
"data": map[string]any{"eligiblePeerCount": peerCount, "combinedPeerSessionCount": peerSessions, "metrics": peerMetrics}}
}
}
s.mu.Lock()
s.runs[key] = storedRun{merchantID: merchantID, run: run}
s.runs[run.RunID] = storedRun{merchantID: merchantID, run: run}
s.mu.Unlock()
return run, nil
}
func peerComparisonSupported(terminalID string) bool { return terminalID == "" }
func (s *Service) Run(merchantID int64, runID string) (Run, bool) {
s.mu.RLock()
stored, ok := s.runs[runID]
s.mu.RUnlock()
return stored.run, ok && stored.merchantID == merchantID
}
type metricValues struct {
total, paid, verified, unverified, failed, reversed, noAttempt, reachedBank, retried, recovered int64
paidAmount, feeAmount int64
initP50, initP95, verifyP50, verifyP95 *float64
initSamples, verifySamples int64
}
func (s *Service) metrics(ctx context.Context, merchantID int64, p Period, terminalID string) (metricValues, error) {
var m metricValues
query := `WITH scoped AS (
SELECT ps.*,count(pa.id) attempts,bool_or(pa.status='InBank') reached_bank
FROM payment_sessions ps LEFT JOIN payment_attempts pa ON pa.session_id=ps.id
WHERE ps.merchant_id=$1 AND ps.created_at >= $2 AND ps.created_at < $3
AND ($4='' OR ps.terminal_id=(SELECT id FROM terminals WHERE merchant_id=$1 AND source_key=$4)) GROUP BY ps.id
), timings AS (
SELECT percentile_disc(0.5) within group(order by pa.init_time_ms)::float8 init_p50,
percentile_disc(0.95) within group(order by pa.init_time_ms)::float8 init_p95,
percentile_disc(0.5) within group(order by pa.verify_time_ms)::float8 verify_p50,
percentile_disc(0.95) within group(order by pa.verify_time_ms)::float8 verify_p95,
count(pa.init_time_ms) init_samples,count(pa.verify_time_ms) verify_samples
FROM payment_attempts pa JOIN payment_sessions ps ON ps.id=pa.session_id
WHERE ps.merchant_id=$1 AND ps.created_at >= $2 AND ps.created_at < $3
AND ($4='' OR ps.terminal_id=(SELECT id FROM terminals WHERE merchant_id=$1 AND source_key=$4)))
SELECT count(*),count(*) filter(where status in ('Paid','Verified')),
count(*) filter(where status='Verified'),count(*) filter(where status='Paid'),count(*) filter(where status='Failed'),
count(*) filter(where status='Reversed'),count(*) filter(where attempts=0),count(*) filter(where reached_bank),
count(*) filter(where attempts>1),count(*) filter(where attempts>1 and status in ('Paid','Verified')),
coalesce(sum(amount_rial) filter(where status in ('Paid','Verified')),0),coalesce(sum(fee_amount_rial) filter(where status in ('Paid','Verified')),0),
t.init_p50,t.init_p95,t.verify_p50,t.verify_p95,t.init_samples,t.verify_samples
FROM scoped CROSS JOIN timings t GROUP BY t.init_p50,t.init_p95,t.verify_p50,t.verify_p95,t.init_samples,t.verify_samples`
err := s.pool.QueryRow(ctx, query, merchantID, p.StartAt, p.EndAt, terminalID).Scan(&m.total, &m.paid, &m.verified, &m.unverified,
&m.failed, &m.reversed, &m.noAttempt, &m.reachedBank, &m.retried, &m.recovered, &m.paidAmount,
&m.feeAmount, &m.initP50, &m.initP95, &m.verifyP50, &m.verifyP95, &m.initSamples, &m.verifySamples)
if err != nil {
return metricValues{}, fmt.Errorf("calculating merchant metrics: %w", err)
}
return m, nil
}
func integerMetric(id string, value int64, unit string) map[string]any {
return map[string]any{"metricId": id, "value": value, "unit": unit}
}
func decimalMetric(id string, numerator, denominator int64, unit string) map[string]any {
var value any
if denominator > 0 {
value = fmt.Sprintf("%.2f", float64(numerator)*100/float64(denominator))
}
return map[string]any{"metricId": id, "value": value, "unit": unit, "numerator": numerator, "denominator": denominator}
}
func unavailableComparison(period Period) map[string]any {
return map[string]any{"state": "UNAVAILABLE", "baselinePeriod": period, "absoluteChange": nil, "relativeChangePercent": nil, "unavailableReason": "SOURCE_UNAVAILABLE", "isActionableInsightEligible": false}
}
func (m metricValues) overview(period Period, history []PaymentHistoryPoint) map[string]any {
return map[string]any{
"bankPaidSessions": integerMetric("bank_paid_sessions", m.paid, "SESSIONS"), "bankPaidAmountRial": integerMetric("bank_paid_amount_rial", m.paidAmount, "RIAL"),
"paymentSuccessRate": decimalMetric("payment_success_rate", m.paid, m.total, "PERCENT"), "verificationCompletionRate": decimalMetric("verification_completion_rate", m.verified, m.paid, "PERCENT"),
"comparisons": map[string]any{"bank_paid_sessions": unavailableComparison(period), "bank_paid_amount_rial": unavailableComparison(period), "payment_success_rate": unavailableComparison(period), "verification_completion_rate": unavailableComparison(period)},
"paymentHistory": history,
}
}
func (s *Service) paymentHistory(ctx context.Context, merchantID int64, period Period, terminalID string) ([]PaymentHistoryPoint, error) {
rows, err := s.pool.Query(ctx, `SELECT (ps.created_at AT TIME ZONE 'Asia/Tehran')::date local_date,
count(*),count(*) FILTER (WHERE ps.status IN ('Paid','Verified')),
coalesce(sum(ps.amount_rial) FILTER (WHERE ps.status IN ('Paid','Verified')),0)
FROM payment_sessions ps WHERE ps.merchant_id=$1 AND ps.created_at >= $2 AND ps.created_at < $3
AND ($4='' OR ps.terminal_id=(SELECT id FROM terminals WHERE merchant_id=$1 AND source_key=$4))
GROUP BY local_date ORDER BY local_date`, merchantID, period.StartAt, period.EndAt, terminalID)
if err != nil {
return nil, fmt.Errorf("querying payment history: %w", err)
}
defer rows.Close()
points := make([]PaymentHistoryPoint, 0, 93)
for rows.Next() {
var date time.Time
var point PaymentHistoryPoint
if err := rows.Scan(&date, &point.TotalSessions, &point.BankPaidSessions, &point.BankPaidAmountRial); err != nil {
return nil, fmt.Errorf("scanning payment history: %w", err)
}
point.LocalDate = date.Format(time.DateOnly)
points = append(points, point)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating payment history: %w", err)
}
return points, nil
}
func timing(id string, p50, p95 *float64, samples, eligible int64) map[string]any {
var a, b any
if p50 != nil {
a = int64(math.Round(*p50))
}
if p95 != nil {
b = int64(math.Round(*p95))
}
var coverage any
if eligible > 0 {
coverage = fmt.Sprintf("%.2f", float64(samples)*100/float64(eligible))
}
return map[string]any{"metricId": id, "p50Ms": a, "p95Ms": b, "sampleSize": samples, "eligibleSampleSize": eligible, "coveragePercent": coverage}
}
func (m metricValues) performance() map[string]any {
return map[string]any{
"bankPaidSessions": integerMetric("bank_paid_sessions", m.paid, "SESSIONS"), "verifiedSessions": integerMetric("verified_sessions", m.verified, "SESSIONS"), "unverifiedPaidSessions": integerMetric("unverified_paid_sessions", m.unverified, "SESSIONS"),
"failedSessions": integerMetric("failed_sessions", m.failed, "SESSIONS"), "reversedSessions": integerMetric("reversed_sessions", m.reversed, "SESSIONS"), "noAttemptSessions": integerMetric("no_attempt_sessions", m.noAttempt, "SESSIONS"),
"reachedBankSessions": integerMetric("reached_bank_sessions", m.reachedBank, "SESSIONS"), "retriedSessions": integerMetric("retried_sessions", m.retried, "SESSIONS"), "retryRecoveryRate": decimalMetric("retry_recovery_rate", m.recovered, m.retried, "PERCENT"),
"initTiming": timing("init_time_ms", m.initP50, m.initP95, m.initSamples, m.total), "verifyTiming": timing("verify_time_ms", m.verifyP50, m.verifyP95, m.verifySamples, m.paid),
}
}
func (m metricValues) fees(period Period) map[string]any {
return map[string]any{"paidFeeAmountRial": integerMetric("paid_fee_amount_rial", m.feeAmount, "RIAL"), "bankPaidAmountRial": integerMetric("bank_paid_amount_rial", m.paidAmount, "RIAL"), "paidFeeRate": decimalMetric("paid_fee_rate", m.feeAmount, m.paidAmount, "RATIO"), "comparison": unavailableComparison(period)}
}
func (s *Service) peerEligibility(ctx context.Context, merchantID int64, p Period) (int64, int64, error) {
var peers, sessions int64
err := s.pool.QueryRow(ctx, `SELECT count(distinct ps.merchant_id),count(*) FROM payment_sessions ps JOIN merchants m ON m.id=ps.merchant_id
WHERE m.category_id=(SELECT category_id FROM merchants WHERE id=$1) AND ps.merchant_id<>$1 AND ps.created_at>=$2 AND ps.created_at<$3`, merchantID, p.StartAt, p.EndAt).Scan(&peers, &sessions)
if err != nil {
return 0, 0, fmt.Errorf("calculating peer eligibility: %w", err)
}
return peers, sessions, nil
}
type peerRow struct {
merchantID int64
values [8]*float64
}
func (s *Service) peerMetrics(ctx context.Context, merchantID int64, p Period) ([]map[string]any, error) {
query := `WITH session_stats AS (SELECT ps.merchant_id,ps.id,ps.status,ps.amount_rial,ps.fee_amount_rial,count(pa.id) attempts
FROM payment_sessions ps LEFT JOIN payment_attempts pa ON pa.session_id=ps.id JOIN merchants m ON m.id=ps.merchant_id
WHERE m.category_id=(SELECT category_id FROM merchants WHERE id=$1) AND ps.created_at>=$2 AND ps.created_at<$3 GROUP BY ps.id),
aggregate AS (SELECT merchant_id,100.0*count(*) filter(where status in ('Paid','Verified'))/nullif(count(*),0) success,
100.0*count(*) filter(where status='Verified')/nullif(count(*) filter(where status in ('Paid','Verified')),0) verification,
100.0*count(*) filter(where attempts>1 and status in ('Paid','Verified'))/nullif(count(*) filter(where attempts>1),0) recovery,
100.0*sum(fee_amount_rial) filter(where status in ('Paid','Verified'))/nullif(sum(amount_rial) filter(where status in ('Paid','Verified')),0) fee FROM session_stats GROUP BY merchant_id),
timing AS (SELECT ps.merchant_id,percentile_disc(.5) within group(order by pa.init_time_ms)::float8 ip50,percentile_disc(.95) within group(order by pa.init_time_ms)::float8 ip95,
percentile_disc(.5) within group(order by pa.verify_time_ms)::float8 vp50,percentile_disc(.95) within group(order by pa.verify_time_ms)::float8 vp95
FROM payment_sessions ps JOIN payment_attempts pa ON pa.session_id=ps.id JOIN merchants m ON m.id=ps.merchant_id WHERE m.category_id=(SELECT category_id FROM merchants WHERE id=$1) AND ps.created_at>=$2 AND ps.created_at<$3 GROUP BY ps.merchant_id)
SELECT a.merchant_id,a.success::float8,a.verification::float8,a.recovery::float8,t.ip50,t.ip95,t.vp50,t.vp95,a.fee::float8 FROM aggregate a JOIN timing t USING(merchant_id)`
rows, err := s.pool.Query(ctx, query, merchantID, p.StartAt, p.EndAt)
if err != nil {
return nil, fmt.Errorf("calculating peer metrics: %w", err)
}
defer rows.Close()
all := []peerRow{}
for rows.Next() {
var row peerRow
if err := rows.Scan(&row.merchantID, &row.values[0], &row.values[1], &row.values[2], &row.values[3], &row.values[4], &row.values[5], &row.values[6], &row.values[7]); err != nil {
return nil, fmt.Errorf("scanning peer metrics: %w", err)
}
all = append(all, row)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating peer metrics: %w", err)
}
ids := []string{"payment_success_rate", "verification_completion_rate", "retry_recovery_rate", "init_time_ms_p50", "init_time_ms_p95", "verify_time_ms_p50", "verify_time_ms_p95", "paid_fee_rate"}
metrics := make([]map[string]any, 0, 8)
for index, id := range ids {
active := 0.0
peers := []float64{}
for _, row := range all {
if row.values[index] == nil {
continue
}
if row.merchantID == merchantID {
active = *row.values[index]
} else {
peers = append(peers, *row.values[index])
}
}
if len(peers) == 0 {
peers = append(peers, 0)
}
slices.Sort(peers)
median := peers[len(peers)/2]
if len(peers)%2 == 0 {
median = (peers[len(peers)/2-1] + median) / 2
}
lowerIsBetter := index >= 3
better := 0
for _, value := range peers {
if (!lowerIsBetter && active >= value) || (lowerIsBetter && active <= value) {
better++
}
}
percentile := float64(better) * 100 / float64(len(peers))
direction := "HIGHER_IS_BETTER"
if lowerIsBetter {
direction = "LOWER_IS_BETTER"
}
metrics = append(metrics, map[string]any{"metricId": id, "cohortMedian": fmt.Sprintf("%.2f", median), "merchantPerformancePercentile": fmt.Sprintf("%.2f", percentile), "direction": direction})
}
return metrics, nil
}

View File

@@ -0,0 +1,13 @@
package analytics
import "testing"
func TestPeerComparisonSupported(t *testing.T) {
t.Parallel()
if peerComparisonSupported("T318") {
t.Fatal("peerComparisonSupported() = true for terminal scope")
}
if !peerComparisonSupported("") {
t.Fatal("peerComparisonSupported() = false for merchant scope")
}
}

View File

@@ -0,0 +1,54 @@
package analytics
import (
"context"
"errors"
"fmt"
)
var ErrUnknownTerminal = errors.New("analytics: unknown terminal")
type Terminal struct {
TerminalID string `json:"terminalId"`
DisplayName string `json:"displayName"`
SessionCount int64 `json:"sessionCount"`
IsDefault bool `json:"isDefault"`
}
func (s *Service) Terminals(ctx context.Context, merchantID int64) ([]Terminal, error) {
rows, err := s.pool.Query(ctx, `SELECT t.source_key,count(ps.id) FROM terminals t
LEFT JOIN payment_sessions ps ON ps.terminal_id=t.id
WHERE t.merchant_id=$1 GROUP BY t.id ORDER BY count(ps.id) DESC,t.source_key`, merchantID)
if err != nil {
return nil, fmt.Errorf("querying merchant terminals: %w", err)
}
defer rows.Close()
terminals := make([]Terminal, 0, 8)
for rows.Next() {
var terminal Terminal
if err := rows.Scan(&terminal.TerminalID, &terminal.SessionCount); err != nil {
return nil, fmt.Errorf("scanning merchant terminal: %w", err)
}
terminal.DisplayName = terminalDisplayName(terminal.TerminalID)
terminal.IsDefault = len(terminals) == 0
terminals = append(terminals, terminal)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating merchant terminals: %w", err)
}
return terminals, nil
}
func (s *Service) validateTerminal(ctx context.Context, merchantID int64, terminalID string) error {
if terminalID == "" {
return nil
}
var exists bool
if err := s.pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM terminals WHERE merchant_id=$1 AND source_key=$2)`, merchantID, terminalID).Scan(&exists); err != nil {
return fmt.Errorf("validating merchant terminal: %w", err)
}
if !exists {
return ErrUnknownTerminal
}
return nil
}

545
internal/api/server.go Normal file
View File

@@ -0,0 +1,545 @@
package api
import (
"bytes"
"context"
"crypto/rand"
"encoding/csv"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"strconv"
"strings"
"time"
"unicode"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/analytics"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/identity"
)
const maxExportBytes = 50 << 20
var errExportBytesExceeded = errors.New("export exceeds byte limit")
type Server struct {
identity *identity.Service
analytics *analytics.Service
logger *slog.Logger
frontendOrigin string
cookieSecure bool
}
type sessionKey struct{}
type tokenKey struct{}
func NewServer(identityService *identity.Service, analyticsService *analytics.Service, logger *slog.Logger, frontendOrigin string, cookieSecure bool) *Server {
return &Server{identity: identityService, analytics: analyticsService, logger: logger, frontendOrigin: frontendOrigin, cookieSecure: cookieSecure}
}
func (s *Server) Handler() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", s.health)
mux.HandleFunc("POST /api/v1/auth/sessions", s.login)
mux.Handle("GET /api/v1/auth/session", s.authenticate(http.HandlerFunc(s.currentSession)))
mux.Handle("DELETE /api/v1/auth/session", s.authenticate(s.csrf(http.HandlerFunc(s.logout))))
mux.Handle("GET /api/v1/analytics/periods", s.authenticate(http.HandlerFunc(s.periods)))
mux.Handle("GET /api/v1/analytics/terminals", s.authenticate(http.HandlerFunc(s.terminals)))
mux.Handle("POST /api/v1/analytics/runs", s.authenticate(s.csrf(http.HandlerFunc(s.startRun))))
mux.Handle("GET /api/v1/analytics/runs/{runId}", s.authenticate(http.HandlerFunc(s.getRun)))
mux.Handle("GET /api/v1/analytics/runs/{runId}/events", s.authenticate(http.HandlerFunc(s.events)))
mux.Handle("GET /api/v1/analytics/session-details", s.authenticate(http.HandlerFunc(s.details)))
mux.Handle("GET /api/v1/analytics/session-details.csv", s.authenticate(http.HandlerFunc(s.exportCSV)))
mux.Handle("GET /api/v1/analytics/sessions/{sessionId}", s.authenticate(http.HandlerFunc(s.sessionDetail)))
return s.boundary(mux)
}
func (s *Server) boundary(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requestID := r.Header.Get("X-Request-ID")
if requestID == "" {
requestID = randomID()
}
w.Header().Set("X-Request-ID", requestID)
w.Header().Set("Cache-Control", "private, no-store")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("X-Frame-Options", "DENY")
w.Header().Set("Referrer-Policy", "no-referrer")
next.ServeHTTP(w, r)
})
}
func (s *Server) authenticate(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
cookie, err := r.Cookie(s.cookieName())
if err != nil {
s.problem(w, r, http.StatusUnauthorized, "UNAUTHENTICATED", "Authentication required")
return
}
session, err := s.identity.Resolve(r.Context(), cookie.Value)
if errors.Is(err, identity.ErrUnauthenticated) {
s.problem(w, r, http.StatusUnauthorized, "UNAUTHENTICATED", "Authentication required")
return
}
if err != nil {
s.logger.Error("session resolution failed", "error", err)
s.problem(w, r, http.StatusServiceUnavailable, "DEPENDENCY_UNAVAILABLE", "Service unavailable")
return
}
ctx := context.WithValue(r.Context(), sessionKey{}, session)
ctx = context.WithValue(ctx, tokenKey{}, cookie.Value)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
func (s *Server) csrf(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
session := sessionFrom(r.Context())
origin := r.Header.Get("Origin")
if r.Header.Get("X-CSRF-Token") != session.CSRFToken || (origin != "" && origin != s.frontendOrigin) {
s.problem(w, r, http.StatusForbidden, "FORBIDDEN", "Request could not be verified")
return
}
next.ServeHTTP(w, r)
})
}
func (s *Server) login(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); origin != "" && origin != s.frontendOrigin {
s.problem(w, r, http.StatusForbidden, "FORBIDDEN", "Request could not be verified")
return
}
r.Body = http.MaxBytesReader(w, r.Body, 16<<10)
var input struct {
Username string `json:"username"`
Password string `json:"password"`
}
decoder := json.NewDecoder(r.Body)
decoder.DisallowUnknownFields()
if err := decoder.Decode(&input); err != nil || input.Username == "" || input.Password == "" {
s.problem(w, r, http.StatusUnprocessableEntity, "VALIDATION_FAILED", "Invalid login request")
return
}
session, token, err := s.identity.Login(r.Context(), input.Username, input.Password)
if errors.Is(err, identity.ErrUnauthenticated) {
s.problem(w, r, http.StatusUnauthorized, "UNAUTHENTICATED", "Invalid username or password")
return
}
if err != nil {
s.logger.Error("login failed", "error", err)
s.problem(w, r, http.StatusServiceUnavailable, "DEPENDENCY_UNAVAILABLE", "Service unavailable")
return
}
http.SetCookie(w, &http.Cookie{Name: s.cookieName(), Value: token, Path: "/", HttpOnly: true, Secure: s.cookieSecure, SameSite: http.SameSiteLaxMode, Expires: session.AbsoluteExpiresAt})
s.writeJSON(w, http.StatusCreated, sessionResponse(session))
}
func (s *Server) currentSession(w http.ResponseWriter, r *http.Request) {
s.writeJSON(w, http.StatusOK, sessionResponse(sessionFrom(r.Context())))
}
func (s *Server) logout(w http.ResponseWriter, r *http.Request) {
if err := s.identity.Logout(r.Context(), r.Context().Value(tokenKey{}).(string)); err != nil {
s.logger.Error("logout failed", "error", err)
s.problem(w, r, http.StatusServiceUnavailable, "DEPENDENCY_UNAVAILABLE", "Service unavailable")
return
}
http.SetCookie(w, &http.Cookie{Name: s.cookieName(), Value: "", Path: "/", HttpOnly: true, Secure: s.cookieSecure, SameSite: http.SameSiteLaxMode, MaxAge: -1})
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) periods(w http.ResponseWriter, r *http.Request) {
periods, err := s.analytics.Periods(r.Context())
if err != nil {
s.logger.Error("period discovery failed", "error", err)
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
periods, err = filterPeriodsByType(periods, r.URL.Query().Get("periodType"))
if err != nil {
s.problem(w, r, 422, "VALIDATION_FAILED", "Invalid reporting period type")
return
}
if len(periods) == 0 {
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
defaultPeriodID := periods[0].PeriodID
for _, period := range periods {
if period.IsDefault {
defaultPeriodID = period.PeriodID
break
}
}
s.writeJSON(w, http.StatusOK, map[string]any{"defaultPeriodId": defaultPeriodID, "periods": periods})
}
func (s *Server) terminals(w http.ResponseWriter, r *http.Request) {
terminals, err := s.analytics.Terminals(r.Context(), sessionFrom(r.Context()).MerchantID)
if err != nil {
s.logger.Error("terminal discovery failed", "error", err)
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
s.writeJSON(w, http.StatusOK, map[string]any{"terminals": terminals})
}
func (s *Server) startRun(w http.ResponseWriter, r *http.Request) {
var input struct {
PeriodID string `json:"periodId"`
TerminalID string `json:"terminalId"`
}
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 16<<10))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&input); err != nil {
s.problem(w, r, 422, "VALIDATION_FAILED", "Invalid analytics request")
return
}
periods, err := s.analytics.Periods(r.Context())
if err != nil {
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
var period analytics.Period
found := false
for _, candidate := range periods {
if candidate.PeriodID == input.PeriodID {
period = candidate
found = true
break
}
}
if !found {
s.problem(w, r, 422, "VALIDATION_FAILED", "Unknown reporting period")
return
}
session := sessionFrom(r.Context())
run, err := s.analytics.Start(r.Context(), session.MerchantID, period, input.TerminalID)
if errors.Is(err, analytics.ErrUnknownTerminal) {
s.problem(w, r, 422, "VALIDATION_FAILED", "Unknown terminal")
return
}
if err != nil {
s.logger.Error("analytics calculation failed", "error", err)
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
w.Header().Set("Location", "/api/v1/analytics/runs/"+run.RunID)
s.writeJSON(w, http.StatusOK, run)
}
func (s *Server) getRun(w http.ResponseWriter, r *http.Request) {
run, ok := s.analytics.Run(sessionFrom(r.Context()).MerchantID, r.PathValue("runId"))
if !ok {
s.problem(w, r, 404, "RESOURCE_NOT_FOUND", "Resource not found")
return
}
s.writeJSON(w, 200, run)
}
func (s *Server) events(w http.ResponseWriter, r *http.Request) {
run, ok := s.analytics.Run(sessionFrom(r.Context()).MerchantID, r.PathValue("runId"))
if !ok {
s.problem(w, r, 404, "RESOURCE_NOT_FOUND", "Resource not found")
return
}
w.Header().Set("Content-Type", "text/event-stream; charset=utf-8")
w.Header().Set("Cache-Control", "no-cache, no-transform")
w.Header().Set("X-Accel-Buffering", "no")
eventID := "evt_" + randomID()
data := map[string]any{"eventId": eventID, "runId": run.RunID, "occurredAt": time.Now().UTC(), "recovery": "RESET_TO_SNAPSHOT", "run": run}
payload, _ := json.Marshal(data)
fmt.Fprintf(w, "id: %s\nevent: run.snapshot\ndata: %s\n\n", eventID, payload)
}
func (s *Server) details(w http.ResponseWriter, r *http.Request) {
period, ok := s.requestPeriod(r)
if !ok {
s.problem(w, r, 422, "VALIDATION_FAILED", "Unknown reporting period")
return
}
limit := 25
if raw := r.URL.Query().Get("limit"); raw != "" {
parsed, err := strconv.Atoi(raw)
if err != nil || parsed < 1 || parsed > 100 {
s.problem(w, r, 422, "VALIDATION_FAILED", "Invalid page limit")
return
}
limit = parsed
}
filter, err := detailFilterFromRequest(r, limit)
if err != nil {
s.problem(w, r, 422, "VALIDATION_FAILED", "Invalid detail request")
return
}
items, next, err := s.analytics.Details(r.Context(), sessionFrom(r.Context()).MerchantID, period, filter)
if errors.Is(err, analytics.ErrInvalidCursor) {
s.problem(w, r, 400, "INVALID_CURSOR", "Invalid detail cursor")
return
}
if err != nil {
s.logger.Error("detail query failed", "error", err)
s.problem(w, r, 422, "VALIDATION_FAILED", "Invalid detail request")
return
}
var nextValue any
if next != "" {
nextValue = next
}
s.writeJSON(w, 200, map[string]any{"period": period, "view": filter.View, "appliedFilters": map[string]any{
"terminalId": nullable(filter.TerminalID), "failureStage": nullable(filter.FailureStage), "status": nullable(filter.Status),
"minAmountRial": filter.MinAmountRial, "maxAmountRial": filter.MaxAmountRial,
}, "items": items, "pageInfo": map[string]any{"limit": limit, "hasNextPage": next != "", "nextCursor": nextValue}})
}
func (s *Server) sessionDetail(w http.ResponseWriter, r *http.Request) {
detail, found, err := s.analytics.Session(r.Context(), sessionFrom(r.Context()).MerchantID, r.PathValue("sessionId"))
if err != nil {
s.logger.Error("session detail query failed", "error", err)
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
if !found {
s.problem(w, r, 404, "RESOURCE_NOT_FOUND", "Resource not found")
return
}
s.writeJSON(w, 200, detail)
}
func (s *Server) exportCSV(w http.ResponseWriter, r *http.Request) {
period, ok := s.requestPeriod(r)
if !ok {
s.problem(w, r, 422, "VALIDATION_FAILED", "Unknown reporting period")
return
}
filter, err := detailFilterFromRequest(r, 100)
if err != nil {
s.problem(w, r, 422, "VALIDATION_FAILED", "Invalid export request")
return
}
exportContext, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
items, err := s.analytics.ExportDetails(exportContext, sessionFrom(r.Context()).MerchantID, period, filter)
if errors.Is(err, analytics.ErrExportTooLarge) {
s.problem(w, r, 413, "EXPORT_TOO_LARGE", "Export exceeds the allowed row limit")
return
}
if errors.Is(err, context.DeadlineExceeded) {
s.problem(w, r, 504, "EXPORT_TIMEOUT", "Export timed out")
return
}
if err != nil {
s.logger.Error("session export failed", "error", err)
s.problem(w, r, 503, "ANALYTICS_UNAVAILABLE", "Analytics unavailable")
return
}
encoded, err := renderExportCSV(items, maxExportBytes)
if errors.Is(err, errExportBytesExceeded) {
s.problem(w, r, 413, "EXPORT_TOO_LARGE", "Export exceeds the allowed byte limit")
return
}
if err != nil {
s.logger.Error("session export encoding failed", "error", err)
s.problem(w, r, 500, "INTERNAL_ERROR", "Export could not be encoded")
return
}
w.Header().Set("Content-Type", "text/csv; charset=utf-8")
filename := fmt.Sprintf("%s-%s-%s.csv", strings.ToLower(filter.View), period.PeriodID, time.Now().UTC().Format("20060102T150405Z"))
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%q; filename*=UTF-8''%s", filename, filename))
if _, err := w.Write(encoded); err != nil {
s.logger.Warn("session export response interrupted", "error", err)
}
}
func renderExportCSV(items []analytics.ExportDetail, maxBytes int64) ([]byte, error) {
var buffer bytes.Buffer
limited := &limitedWriter{writer: &buffer, remaining: maxBytes}
if _, err := limited.Write([]byte{0xEF, 0xBB, 0xBF}); err != nil {
return nil, err
}
writer := csv.NewWriter(limited)
writer.UseCRLF = true
if err := writer.Write([]string{"شناسه نشست پرداخت", "شناسه پایانه", "وضعیت پرداخت", "مبلغ ریال", "کارمزد ریال", "زمان ایجاد", "زمان انقضا", "تعداد تلاش", "وضعیت آخرین تلاش", "زمان آخرین تلاش", "زمان تأیید", "زمان تسویه", "مرحله مشاهده شده شکست"}); err != nil {
return nil, err
}
for _, item := range items {
lastStatus, lastAt := "", ""
if item.LastAttemptStatus != nil {
lastStatus = persianStatus(*item.LastAttemptStatus)
}
if item.LastAttemptAt != nil {
lastAt = item.LastAttemptAt.UTC().Format(time.RFC3339)
}
if err := writer.Write([]string{spreadsheetSafeText(item.SessionID), spreadsheetSafeText(item.TerminalID), spreadsheetSafeText(persianStatus(item.SessionStatus)), strconv.FormatInt(item.AmountRial, 10), strconv.FormatInt(item.FeeAmountRial, 10), item.CreatedAt.UTC().Format(time.RFC3339), formatTime(item.ExpiresAt), strconv.Itoa(item.AttemptCount), spreadsheetSafeText(lastStatus), lastAt, formatTime(item.VerifiedAt), formatTime(item.SettledAt), spreadsheetSafeText(stringValue(item.FailureStage))}); err != nil {
return nil, err
}
}
writer.Flush()
if err := writer.Error(); err != nil {
return nil, err
}
return buffer.Bytes(), nil
}
type limitedWriter struct {
writer io.Writer
remaining int64
}
func (w *limitedWriter) Write(data []byte) (int, error) {
if int64(len(data)) > w.remaining {
return 0, errExportBytesExceeded
}
written, err := w.writer.Write(data)
w.remaining -= int64(written)
return written, err
}
func spreadsheetSafeText(value string) string {
trimmed := strings.TrimLeftFunc(value, unicode.IsSpace)
if value == "" || trimmed == "" {
return value
}
first := trimmed[0]
if first == '=' || first == '+' || first == '-' || first == '@' || value[0] == '\t' || value[0] == '\r' || value[0] == '\n' {
return "'" + value
}
return value
}
func (s *Server) requestPeriod(r *http.Request) (analytics.Period, bool) {
periods, err := s.analytics.Periods(r.Context())
if err != nil {
return analytics.Period{}, false
}
id := r.URL.Query().Get("periodId")
for _, p := range periods {
if p.PeriodID == id {
return p, true
}
}
return analytics.Period{}, false
}
func nullable(value string) any {
if value == "" {
return nil
}
return value
}
func optionalNonNegativeInt64(value string) (*int64, bool) {
if value == "" {
return nil, true
}
parsed, err := strconv.ParseInt(value, 10, 64)
if err != nil || parsed < 0 {
return nil, false
}
return &parsed, true
}
func detailFilterFromRequest(r *http.Request, limit int) (analytics.DetailFilter, error) {
minimum, valid := optionalNonNegativeInt64(r.URL.Query().Get("minAmountRial"))
if !valid {
return analytics.DetailFilter{}, errors.New("invalid minimum amount")
}
maximum, valid := optionalNonNegativeInt64(r.URL.Query().Get("maxAmountRial"))
if !valid {
return analytics.DetailFilter{}, errors.New("invalid maximum amount")
}
filter := analytics.DetailFilter{
View: r.URL.Query().Get("view"), FailureStage: r.URL.Query().Get("failureStage"),
TerminalID: r.URL.Query().Get("terminalId"), Status: r.URL.Query().Get("status"),
MinAmountRial: minimum, MaxAmountRial: maximum, Cursor: r.URL.Query().Get("cursor"), Limit: limit,
}
if err := filter.Validate(); err != nil {
return analytics.DetailFilter{}, err
}
return filter, nil
}
func filterPeriodsByType(periods []analytics.Period, requested string) ([]analytics.Period, error) {
if requested == "" {
return periods, nil
}
allowed := map[string]bool{"DAY": true, "WEEK": true, "JALALI_MONTH": true, "PERSIAN_SEASON": true}
if !allowed[requested] {
return nil, errors.New("invalid reporting period type")
}
filtered := make([]analytics.Period, 0, 1)
for _, period := range periods {
if period.PeriodType == requested {
period.IsDefault = len(filtered) == 0
filtered = append(filtered, period)
}
}
return filtered, nil
}
func formatTime(value *time.Time) string {
if value == nil {
return ""
}
return value.UTC().Format(time.RFC3339)
}
func stringValue(value *string) string {
if value == nil {
return ""
}
return *value
}
func persianStatus(value string) string {
switch value {
case "Paid":
return "پرداخت شده و تأیید نشده"
case "Verified":
return "تأیید شده"
case "Failed":
return "ناموفق"
case "Reversed":
return "برگشت خورده"
case "InBank":
return "در بانک"
default:
return value
}
}
func (s *Server) health(w http.ResponseWriter, r *http.Request) {
s.writeJSON(w, 200, map[string]string{"status": "ok"})
}
func (s *Server) cookieName() string {
if s.cookieSecure {
return "__Host-zpa_session"
}
return "zpa_session"
}
func sessionFrom(ctx context.Context) identity.Session {
return ctx.Value(sessionKey{}).(identity.Session)
}
func sessionResponse(v identity.Session) map[string]any {
return map[string]any{"user": map[string]any{"userId": fmt.Sprintf("user_%d", v.UserID), "username": v.Username}, "merchant": map[string]any{"merchantId": fmt.Sprintf("merchant_%d", v.MerchantID), "displayName": v.MerchantDisplayName, "profileIsSynthetic": v.ProfileIsSynthetic}, "createdAt": v.CreatedAt, "idleExpiresAt": v.IdleExpiresAt, "absoluteExpiresAt": v.AbsoluteExpiresAt, "csrfToken": v.CSRFToken}
}
func (s *Server) writeJSON(w http.ResponseWriter, status int, value any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if err := json.NewEncoder(w).Encode(value); err != nil {
s.logger.Error("response encoding failed", "error", err)
}
}
func (s *Server) problem(w http.ResponseWriter, r *http.Request, status int, code, detail string) {
w.Header().Set("Content-Type", "application/problem+json")
w.WriteHeader(status)
value := map[string]any{"type": "https://zarinpal.amiirkhl.ir/docs/errors/" + strings.ToLower(strings.ReplaceAll(code, "_", "-")), "title": http.StatusText(status), "status": status, "code": code, "detail": detail, "instance": r.URL.Path, "requestId": w.Header().Get("X-Request-ID")}
if err := json.NewEncoder(w).Encode(value); err != nil {
s.logger.Error("problem encoding failed", "error", err)
}
}
func randomID() string {
data := make([]byte, 12)
if _, err := rand.Read(data); err != nil {
return fmt.Sprint(time.Now().UnixNano())
}
return hex.EncodeToString(data)
}

125
internal/api/server_test.go Normal file
View File

@@ -0,0 +1,125 @@
package api
import (
"errors"
"net/http/httptest"
"strings"
"testing"
"time"
"git.amiirkhl.ir/starcoach-hackathon/zarinpal-analytics-backend/internal/analytics"
)
func TestServerCookieName(t *testing.T) {
t.Parallel()
tests := []struct {
name string
secure bool
expected string
}{
{name: "local development", secure: false, expected: "zpa_session"},
{name: "secure deployment", secure: true, expected: "__Host-zpa_session"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
server := &Server{cookieSecure: tt.secure}
if got := server.cookieName(); got != tt.expected {
t.Fatalf("cookieName() = %q, want %q", got, tt.expected)
}
})
}
}
func TestRenderExportCSVIsBoundedAndFormulaSafe(t *testing.T) {
t.Parallel()
item := analytics.ExportDetail{Detail: analytics.Detail{
SessionID: "=session", TerminalID: "+terminal", SessionStatus: "Verified",
AmountRial: 1000, FeeAmountRial: 10, CreatedAt: time.Date(2026, time.June, 20, 8, 0, 0, 0, time.UTC),
}}
encoded, err := renderExportCSV([]analytics.ExportDetail{item}, 4096)
if err != nil {
t.Fatal(err)
}
text := string(encoded)
if !strings.Contains(text, "'=session") || !strings.Contains(text, "'+terminal") {
t.Fatalf("renderExportCSV() did not neutralize formulas: %q", text)
}
if _, err := renderExportCSV([]analytics.ExportDetail{item}, 10); !errors.Is(err, errExportBytesExceeded) {
t.Fatalf("renderExportCSV() error = %v, want errExportBytesExceeded", err)
}
}
func TestPersianStatus(t *testing.T) {
t.Parallel()
if got := persianStatus("Verified"); got != "تأیید شده" {
t.Fatalf("persianStatus() = %q", got)
}
}
func TestOptionalNonNegativeInt64(t *testing.T) {
t.Parallel()
tests := []struct {
name, input string
want *int64
valid bool
}{
{name: "omitted", valid: true},
{name: "zero", input: "0", want: int64Pointer(0), valid: true},
{name: "positive", input: "125000", want: int64Pointer(125000), valid: true},
{name: "negative", input: "-1"},
{name: "not a number", input: "یک"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, valid := optionalNonNegativeInt64(tt.input)
if valid != tt.valid || !equalInt64Pointers(got, tt.want) {
t.Fatalf("optionalNonNegativeInt64(%q) = %v,%v, want %v,%v", tt.input, got, valid, tt.want, tt.valid)
}
})
}
}
func int64Pointer(value int64) *int64 { return &value }
func equalInt64Pointers(left, right *int64) bool {
if left == nil || right == nil {
return left == right
}
return *left == *right
}
func TestDetailFilterFromRequestParsesCanonicalFilters(t *testing.T) {
t.Parallel()
request := httptest.NewRequest("GET", "/?view=ALL&failureStage=&terminalId=T318&status=Verified&minAmountRial=1000&maxAmountRial=9000&cursor=opaque", nil)
filter, err := detailFilterFromRequest(request, 25)
if err != nil {
t.Fatal(err)
}
if filter.View != "ALL" || filter.TerminalID != "T318" || filter.Status != "Verified" || filter.Cursor != "opaque" || filter.Limit != 25 {
t.Fatalf("detailFilterFromRequest() = %+v", filter)
}
if filter.MinAmountRial == nil || *filter.MinAmountRial != 1000 || filter.MaxAmountRial == nil || *filter.MaxAmountRial != 9000 {
t.Fatalf("detailFilterFromRequest() amount bounds = %v/%v", filter.MinAmountRial, filter.MaxAmountRial)
}
}
func TestDetailFilterFromRequestRejectsDescendingAmountRange(t *testing.T) {
t.Parallel()
request := httptest.NewRequest("GET", "/?view=ALL&minAmountRial=9000&maxAmountRial=1000", nil)
if _, err := detailFilterFromRequest(request, 25); err == nil {
t.Fatal("detailFilterFromRequest() error = nil")
}
}
func TestFilterPeriodsByType(t *testing.T) {
t.Parallel()
periods := []analytics.Period{{PeriodType: "DAY"}, {PeriodType: "WEEK"}}
filtered, err := filterPeriodsByType(periods, "DAY")
if err != nil || len(filtered) != 1 || filtered[0].PeriodType != "DAY" || !filtered[0].IsDefault {
t.Fatalf("filterPeriodsByType() = %+v, %v", filtered, err)
}
if _, err := filterPeriodsByType(periods, "MONTH"); err == nil {
t.Fatal("filterPeriodsByType() accepted invalid type")
}
}

74
internal/config/config.go Normal file
View File

@@ -0,0 +1,74 @@
package config
import (
"errors"
"fmt"
"os"
"strconv"
)
type Config struct {
HTTPAddr string
DatabaseURL string
RedisAddr string
RedisPassword string
ImportPath string
DemoUsername string
DemoPassword string
SessionSecret string
CursorSecret string
FrontendOrigin string
CookieSecure bool
}
func Load() (Config, error) {
cfg := Config{
HTTPAddr: value("HTTP_ADDR", "127.0.0.1:8080"),
DatabaseURL: os.Getenv("DATABASE_URL"),
RedisAddr: value("REDIS_ADDR", "localhost:6379"),
RedisPassword: os.Getenv("REDIS_PASSWORD"),
ImportPath: value("IMPORT_PATH", "data/incoming/challenge_data.csv.gz"),
DemoUsername: value("DEMO_USERNAME", "merchant-0001"),
DemoPassword: os.Getenv("DEMO_PASSWORD"),
SessionSecret: os.Getenv("SESSION_SECRET"),
CursorSecret: os.Getenv("CURSOR_SECRET"),
FrontendOrigin: value("FRONTEND_ORIGIN", "http://localhost:5173"),
}
var err error
cfg.CookieSecure, err = strconv.ParseBool(value("COOKIE_SECURE", "false"))
if err != nil {
return Config{}, fmt.Errorf("parsing cookie secure: %w", err)
}
if cfg.DatabaseURL == "" {
return Config{}, errors.New("config: DATABASE_URL is required")
}
return cfg, nil
}
func (c Config) ValidateAPI() error {
if len(c.SessionSecret) < 32 {
return errors.New("config: SESSION_SECRET must contain at least 32 characters")
}
if len(c.CursorSecret) < 32 {
return errors.New("config: CURSOR_SECRET must contain at least 32 characters")
}
if c.CursorSecret == c.SessionSecret {
return errors.New("config: CURSOR_SECRET must differ from SESSION_SECRET")
}
return nil
}
func (c Config) ValidateSeed() error {
if len(c.DemoPassword) < 12 {
return errors.New("config: DEMO_PASSWORD must contain at least 12 characters")
}
return nil
}
func value(name, fallback string) string {
if value := os.Getenv(name); value != "" {
return value
}
return fallback
}

View File

@@ -0,0 +1,35 @@
package config
import "testing"
func TestLoadRequiresDatabaseURL(t *testing.T) {
t.Setenv("DATABASE_URL", "")
if _, err := Load(); err == nil {
t.Fatal("Load() error = nil, want required database url error")
}
}
func TestValidateAPIRequiresLongSecret(t *testing.T) {
t.Parallel()
cfg := Config{SessionSecret: "short", CursorSecret: "cursor-secret-that-is-at-least-32-bytes"}
if err := cfg.ValidateAPI(); err == nil {
t.Fatal("ValidateAPI() error = nil, want secret length error")
}
}
func TestValidateAPIRequiresLongCursorSecret(t *testing.T) {
t.Parallel()
cfg := Config{SessionSecret: "session-secret-that-is-at-least-32-bytes", CursorSecret: "short"}
if err := cfg.ValidateAPI(); err == nil {
t.Fatal("ValidateAPI() error = nil, want cursor secret length error")
}
}
func TestValidateAPIRequiresIndependentCursorSecret(t *testing.T) {
t.Parallel()
secret := "shared-secret-that-is-at-least-32-characters"
cfg := Config{SessionSecret: secret, CursorSecret: secret}
if err := cfg.ValidateAPI(); err == nil {
t.Fatal("ValidateAPI() error = nil, want independent secrets error")
}
}

View File

@@ -0,0 +1,30 @@
package database
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
func Open(ctx context.Context, databaseURL string) (*pgxpool.Pool, error) {
poolConfig, err := pgxpool.ParseConfig(databaseURL)
if err != nil {
return nil, fmt.Errorf("parsing database configuration: %w", err)
}
poolConfig.MaxConns = 12
poolConfig.MinConns = 1
poolConfig.MaxConnLifetime = 30 * time.Minute
poolConfig.MaxConnIdleTime = 5 * time.Minute
pool, err := pgxpool.NewWithConfig(ctx, poolConfig)
if err != nil {
return nil, fmt.Errorf("opening database: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("pinging database: %w", err)
}
return pool, nil
}

View File

@@ -0,0 +1,23 @@
package database
import (
"context"
"fmt"
"strings"
"github.com/jackc/pgx/v5/pgxpool"
)
// Migrate applies the idempotent local MVP schema. Production migration tooling
// can execute the same reviewed SQL from migrations/.
func Migrate(ctx context.Context, pool *pgxpool.Pool, migration string) error {
for statement := range strings.SplitSeq(migration, "-- statement-break") {
if strings.TrimSpace(statement) == "" {
continue
}
if _, err := pool.Exec(ctx, statement); err != nil {
return fmt.Errorf("applying migration: %w", err)
}
}
return nil
}

View File

@@ -0,0 +1,56 @@
package identity
import (
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"errors"
"fmt"
"strings"
"golang.org/x/crypto/argon2"
)
const (
argonMemory = 64 * 1024
argonIterations = 3
argonParallelism = 2
argonSaltLength = 16
argonKeyLength = 32
)
func HashPassword(password string) (string, error) {
if len(password) < 12 {
return "", errors.New("identity: password must contain at least 12 characters")
}
salt := make([]byte, argonSaltLength)
if _, err := rand.Read(salt); err != nil {
return "", fmt.Errorf("generating password salt: %w", err)
}
hash := argon2.IDKey([]byte(password), salt, argonIterations, argonMemory, argonParallelism, argonKeyLength)
return fmt.Sprintf("$argon2id$v=19$m=%d,t=%d,p=%d$%s$%s", argonMemory, argonIterations,
argonParallelism, base64.RawStdEncoding.EncodeToString(salt), base64.RawStdEncoding.EncodeToString(hash)), nil
}
func VerifyPassword(password, encoded string) bool {
parts := strings.Split(encoded, "$")
if len(parts) != 6 || parts[1] != "argon2id" || parts[2] != "v=19" {
return false
}
var memory uint32
var iterations uint32
var parallelism uint8
if _, err := fmt.Sscanf(parts[3], "m=%d,t=%d,p=%d", &memory, &iterations, &parallelism); err != nil {
return false
}
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
if err != nil {
return false
}
expected, err := base64.RawStdEncoding.DecodeString(parts[5])
if err != nil || len(expected) == 0 {
return false
}
actual := argon2.IDKey([]byte(password), salt, iterations, memory, parallelism, uint32(len(expected)))
return subtle.ConstantTimeCompare(actual, expected) == 1
}

View File

@@ -0,0 +1,24 @@
package identity
import "testing"
func TestHashPasswordAndVerifyPassword(t *testing.T) {
t.Parallel()
hash, err := HashPassword("correct horse battery staple")
if err != nil {
t.Fatal(err)
}
if !VerifyPassword("correct horse battery staple", hash) {
t.Fatal("VerifyPassword() = false, want true")
}
if VerifyPassword("incorrect password", hash) {
t.Fatal("VerifyPassword() = true for incorrect password")
}
}
func TestHashPasswordRejectsShortPassword(t *testing.T) {
t.Parallel()
if _, err := HashPassword("short"); err == nil {
t.Fatal("HashPassword() error = nil, want length error")
}
}

39
internal/identity/seed.go Normal file
View File

@@ -0,0 +1,39 @@
package identity
import (
"context"
"fmt"
"github.com/jackc/pgx/v5/pgxpool"
)
type SeedResult struct {
Username string
MerchantSourceKey string
MerchantName string
}
func SeedDemo(ctx context.Context, pool *pgxpool.Pool, username, password string) (SeedResult, error) {
hash, err := HashPassword(password)
if err != nil {
return SeedResult{}, err
}
var result SeedResult
var merchantID int64
err = pool.QueryRow(ctx, `SELECT m.id,m.source_key,m.display_name
FROM merchants m JOIN payment_sessions ps ON ps.merchant_id=m.id
GROUP BY m.id ORDER BY (count(DISTINCT ps.terminal_id)>1) DESC,count(*) DESC,m.source_key LIMIT 1`).Scan(
&merchantID, &result.MerchantSourceKey, &result.MerchantName,
)
if err != nil {
return SeedResult{}, fmt.Errorf("selecting demo merchant: %w", err)
}
_, err = pool.Exec(ctx, `INSERT INTO merchant_users(merchant_id,username,password_hash)
VALUES ($1,$2,$3) ON CONFLICT (username) DO UPDATE
SET merchant_id=excluded.merchant_id,password_hash=excluded.password_hash,is_active=true`, merchantID, username, hash)
if err != nil {
return SeedResult{}, fmt.Errorf("seeding demo merchant user: %w", err)
}
result.Username = username
return result, nil
}

View File

@@ -0,0 +1,113 @@
package identity
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
var ErrUnauthenticated = errors.New("identity: unauthenticated")
type Session struct {
UserID int64
MerchantID int64
Username string
MerchantSourceKey string
MerchantDisplayName string
ProfileIsSynthetic bool
CreatedAt time.Time
IdleExpiresAt time.Time
AbsoluteExpiresAt time.Time
CSRFToken string
}
type Service struct {
pool *pgxpool.Pool
now func() time.Time
}
func New(pool *pgxpool.Pool) *Service { return &Service{pool: pool, now: time.Now} }
func (s *Service) Login(ctx context.Context, username, password string) (Session, string, error) {
var userID, merchantID int64
var passwordHash, sourceKey, displayName string
var isActive, isSynthetic bool
err := s.pool.QueryRow(ctx, `SELECT u.id,u.merchant_id,u.password_hash,u.is_active,m.source_key,m.display_name,m.profile_is_synthetic
FROM merchant_users u JOIN merchants m ON m.id=u.merchant_id WHERE u.username=$1`, username).Scan(
&userID, &merchantID, &passwordHash, &isActive, &sourceKey, &displayName, &isSynthetic,
)
if err != nil || !isActive || !VerifyPassword(password, passwordHash) {
// Perform an Argon2 calculation for missing users to reduce enumeration timing.
if errors.Is(err, pgx.ErrNoRows) {
_, _ = HashPassword("invalid-credential-padding")
}
return Session{}, "", ErrUnauthenticated
}
token, err := randomToken(32)
if err != nil {
return Session{}, "", err
}
csrf, err := randomToken(32)
if err != nil {
return Session{}, "", err
}
now := s.now().UTC()
session := Session{UserID: userID, MerchantID: merchantID, Username: username, MerchantSourceKey: sourceKey,
MerchantDisplayName: displayName, ProfileIsSynthetic: isSynthetic, CreatedAt: now,
IdleExpiresAt: now.Add(30 * time.Minute), AbsoluteExpiresAt: now.Add(8 * time.Hour), CSRFToken: csrf}
hash := sha256.Sum256([]byte(token))
_, err = s.pool.Exec(ctx, `INSERT INTO auth_sessions(user_id,merchant_id,token_hash,csrf_token,created_at,idle_expires_at,absolute_expires_at,last_seen_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$5)`, userID, merchantID, hash[:], csrf, now, session.IdleExpiresAt, session.AbsoluteExpiresAt)
if err != nil {
return Session{}, "", fmt.Errorf("creating auth session: %w", err)
}
return session, token, nil
}
func (s *Service) Resolve(ctx context.Context, token string) (Session, error) {
if token == "" {
return Session{}, ErrUnauthenticated
}
hash := sha256.Sum256([]byte(token))
var session Session
err := s.pool.QueryRow(ctx, `SELECT u.id,m.id,u.username,m.source_key,m.display_name,m.profile_is_synthetic,
a.created_at,a.idle_expires_at,a.absolute_expires_at,a.csrf_token
FROM auth_sessions a JOIN merchant_users u ON u.id=a.user_id JOIN merchants m ON m.id=a.merchant_id
WHERE a.token_hash=$1 AND a.revoked_at IS NULL AND u.is_active
AND a.idle_expires_at>now() AND a.absolute_expires_at>now()`, hash[:]).Scan(
&session.UserID, &session.MerchantID, &session.Username, &session.MerchantSourceKey,
&session.MerchantDisplayName, &session.ProfileIsSynthetic, &session.CreatedAt,
&session.IdleExpiresAt, &session.AbsoluteExpiresAt, &session.CSRFToken,
)
if errors.Is(err, pgx.ErrNoRows) {
return Session{}, ErrUnauthenticated
}
if err != nil {
return Session{}, fmt.Errorf("resolving auth session: %w", err)
}
return session, nil
}
func (s *Service) Logout(ctx context.Context, token string) error {
hash := sha256.Sum256([]byte(token))
_, err := s.pool.Exec(ctx, `UPDATE auth_sessions SET revoked_at=now() WHERE token_hash=$1 AND revoked_at IS NULL`, hash[:])
if err != nil {
return fmt.Errorf("revoking auth session: %w", err)
}
return nil
}
func randomToken(size int) (string, error) {
data := make([]byte, size)
if _, err := rand.Read(data); err != nil {
return "", fmt.Errorf("generating secure token: %w", err)
}
return base64.RawURLEncoding.EncodeToString(data), nil
}

View File

@@ -0,0 +1,200 @@
package importer
import (
"compress/gzip"
"context"
"crypto/sha256"
"encoding/csv"
"encoding/hex"
"errors"
"fmt"
"io"
"os"
"strings"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
var expectedHeader = []string{
"session_key", "try_seq", "terminal_key", "merchant_key", "category_id",
"category_title", "amount", "adjusted_fee", "session_status", "try_status",
"switch_response_code", "psp_code", "issuer_bank_code", "payer_card_key",
"verify_type", "init_time_ms", "verify_time_ms", "created_at", "try_created_at",
"verified_at", "settled_at", "expire_in",
}
type Result struct {
IsNoop bool
SourceSHA256 string
SourceRows int64
SessionRows int64
AttemptRows int64
}
func Run(ctx context.Context, pool *pgxpool.Pool, path string) (Result, error) {
checksum, err := checksumFile(path)
if err != nil {
return Result{}, err
}
if err := validateHeader(path); err != nil {
return Result{}, err
}
var completed bool
err = pool.QueryRow(ctx, `SELECT EXISTS (
SELECT 1 FROM import_runs WHERE source_sha256=$1 AND status='COMPLETED'
)`, checksum).Scan(&completed)
if err != nil {
return Result{}, fmt.Errorf("checking import manifest: %w", err)
}
if completed {
return Result{IsNoop: true, SourceSHA256: checksum}, nil
}
tx, err := pool.BeginTx(ctx, pgx.TxOptions{})
if err != nil {
return Result{}, fmt.Errorf("starting import transaction: %w", err)
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(943271)`); err != nil {
return Result{}, fmt.Errorf("locking importer: %w", err)
}
var runID int64
err = tx.QueryRow(ctx, `INSERT INTO import_runs(source_sha256,source_path,status)
VALUES ($1,$2,'RUNNING') RETURNING id`, checksum, path).Scan(&runID)
if err != nil {
return Result{}, fmt.Errorf("creating import manifest: %w", err)
}
if err := createStaging(ctx, tx); err != nil {
return Result{}, err
}
file, err := os.Open(path)
if err != nil {
return Result{}, fmt.Errorf("opening import archive: %w", err)
}
defer file.Close()
gzipReader, err := gzip.NewReader(file)
if err != nil {
return Result{}, fmt.Errorf("opening gzip stream: %w", err)
}
defer gzipReader.Close()
copySQL := `COPY import_staging (` + strings.Join(expectedHeader, ",") + `) FROM STDIN WITH (FORMAT csv, HEADER true)`
copyResult, err := tx.Conn().PgConn().CopyFrom(ctx, gzipReader, copySQL)
if err != nil {
return Result{}, fmt.Errorf("copying archive into staging: %w", err)
}
result := Result{SourceSHA256: checksum, SourceRows: copyResult.RowsAffected()}
if err := promote(ctx, tx, runID, &result); err != nil {
return Result{}, err
}
if err := tx.Commit(ctx); err != nil {
return Result{}, fmt.Errorf("committing import: %w", err)
}
return result, nil
}
func checksumFile(path string) (string, error) {
file, err := os.Open(path)
if err != nil {
return "", fmt.Errorf("opening source for checksum: %w", err)
}
defer file.Close()
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
return "", fmt.Errorf("checksumming source: %w", err)
}
return hex.EncodeToString(hash.Sum(nil)), nil
}
func validateHeader(path string) error {
file, err := os.Open(path)
if err != nil {
return fmt.Errorf("opening source for preflight: %w", err)
}
defer file.Close()
gzipReader, err := gzip.NewReader(file)
if err != nil {
return fmt.Errorf("opening source gzip for preflight: %w", err)
}
defer gzipReader.Close()
header, err := csv.NewReader(gzipReader).Read()
if err != nil {
return fmt.Errorf("reading source header: %w", err)
}
if len(header) != len(expectedHeader) {
return fmt.Errorf("source header has %d columns, expected %d", len(header), len(expectedHeader))
}
for i := range expectedHeader {
if header[i] != expectedHeader[i] {
return fmt.Errorf("source header column %d is %q, expected %q", i+1, header[i], expectedHeader[i])
}
}
return nil
}
func createStaging(ctx context.Context, tx pgx.Tx) error {
columns := make([]string, 0, len(expectedHeader))
for _, column := range expectedHeader {
columns = append(columns, pgx.Identifier{column}.Sanitize()+" text")
}
statement := `CREATE TEMP TABLE import_staging (` + strings.Join(columns, ",") + `) ON COMMIT DROP`
if _, err := tx.Exec(ctx, statement); err != nil {
return fmt.Errorf("creating import staging table: %w", err)
}
return nil
}
func promote(ctx context.Context, tx pgx.Tx, runID int64, result *Result) error {
queries := []string{
`INSERT INTO merchants(source_key,category_id,category_title,display_name)
SELECT DISTINCT merchant_key,category_id::bigint,category_title,'کسب‌وکار '||merchant_key
FROM import_staging ON CONFLICT (source_key) DO NOTHING`,
`INSERT INTO terminals(merchant_id,source_key,verify_type)
SELECT DISTINCT m.id,s.terminal_key,s.verify_type FROM import_staging s
JOIN merchants m ON m.source_key=s.merchant_key
ON CONFLICT (merchant_id,source_key) DO NOTHING`,
`INSERT INTO payment_sessions(import_run_id,merchant_id,terminal_id,source_key,status,amount_rial,fee_amount_rial,payer_card_key,created_at,expires_at,verified_at,settled_at)
SELECT DISTINCT ON (m.id,s.session_key) $1,m.id,t.id,s.session_key,s.session_status,s.amount::bigint,s.adjusted_fee::bigint,
nullif(s.payer_card_key,''),s.created_at::timestamp AT TIME ZONE 'Asia/Tehran',
nullif(s.expire_in,'')::timestamp AT TIME ZONE 'Asia/Tehran',nullif(s.verified_at,'')::timestamp AT TIME ZONE 'Asia/Tehran',
nullif(s.settled_at,'')::timestamp AT TIME ZONE 'Asia/Tehran'
FROM import_staging s JOIN merchants m ON m.source_key=s.merchant_key
JOIN terminals t ON t.merchant_id=m.id AND t.source_key=s.terminal_key
ORDER BY m.id,s.session_key,s.try_seq::integer DESC
ON CONFLICT (merchant_id,source_key) DO NOTHING`,
`INSERT INTO payment_attempts(session_id,sequence,status,created_at,init_time_ms,verify_time_ms)
SELECT ps.id,s.try_seq::integer,s.try_status,nullif(s.try_created_at,'')::timestamp AT TIME ZONE 'Asia/Tehran',
nullif(s.init_time_ms,'')::integer,nullif(s.verify_time_ms,'')::integer
FROM import_staging s JOIN merchants m ON m.source_key=s.merchant_key
JOIN payment_sessions ps ON ps.merchant_id=m.id AND ps.source_key=s.session_key
WHERE s.try_seq::integer > 0 AND s.try_status <> 'NoAttempt'
ON CONFLICT (session_id,sequence) DO NOTHING`,
}
for i, query := range queries {
args := []any{}
if i == 2 {
args = append(args, runID)
}
if _, err := tx.Exec(ctx, query, args...); err != nil {
return fmt.Errorf("promoting import phase %d: %w", i+1, err)
}
}
if err := tx.QueryRow(ctx, `SELECT count(*) FROM payment_sessions WHERE import_run_id=$1`, runID).Scan(&result.SessionRows); err != nil {
return fmt.Errorf("counting imported sessions: %w", err)
}
if err := tx.QueryRow(ctx, `SELECT count(*) FROM payment_attempts pa JOIN payment_sessions ps ON ps.id=pa.session_id WHERE ps.import_run_id=$1`, runID).Scan(&result.AttemptRows); err != nil {
return fmt.Errorf("counting imported attempts: %w", err)
}
if result.SessionRows == 0 || result.AttemptRows > result.SourceRows {
return errors.New("import reconciliation failed")
}
_, err := tx.Exec(ctx, `UPDATE import_runs SET status='COMPLETED',source_rows=$2,session_rows=$3,attempt_rows=$4,completed_at=now() WHERE id=$1`, runID, result.SourceRows, result.SessionRows, result.AttemptRows)
if err != nil {
return fmt.Errorf("completing import manifest: %w", err)
}
return nil
}

View File

@@ -0,0 +1,44 @@
package importer
import (
"compress/gzip"
"os"
"path/filepath"
"strings"
"testing"
)
func TestValidateHeader(t *testing.T) {
t.Parallel()
tests := []struct {
name string
header string
wantErr bool
}{
{name: "approved header", header: strings.Join(expectedHeader, ",")},
{name: "unknown column", header: "wrong", wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "input.csv.gz")
file, err := os.Create(path)
if err != nil {
t.Fatal(err)
}
writer := gzip.NewWriter(file)
if _, err := writer.Write([]byte(tt.header + "\n")); err != nil {
t.Fatal(err)
}
if err := writer.Close(); err != nil {
t.Fatal(err)
}
if err := file.Close(); err != nil {
t.Fatal(err)
}
err = validateHeader(path)
if (err != nil) != tt.wantErr {
t.Fatalf("validateHeader() error = %v, wantErr %v", err, tt.wantErr)
}
})
}
}

93
migrations/000001_mvp.sql Normal file
View File

@@ -0,0 +1,93 @@
CREATE TABLE IF NOT EXISTS schema_migrations (
version bigint PRIMARY KEY,
applied_at timestamptz NOT NULL DEFAULT now()
);
-- statement-break
CREATE TABLE IF NOT EXISTS import_runs (
id bigserial PRIMARY KEY,
source_sha256 text NOT NULL UNIQUE,
source_path text NOT NULL,
status text NOT NULL CHECK (status IN ('RUNNING','COMPLETED','FAILED')),
source_rows bigint NOT NULL DEFAULT 0,
session_rows bigint NOT NULL DEFAULT 0,
attempt_rows bigint NOT NULL DEFAULT 0,
started_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz
);
-- statement-break
CREATE TABLE IF NOT EXISTS merchants (
id bigserial PRIMARY KEY,
source_key text NOT NULL UNIQUE,
category_id bigint NOT NULL,
category_title text NOT NULL,
display_name text NOT NULL,
profile_is_synthetic boolean NOT NULL DEFAULT true
);
-- statement-break
CREATE TABLE IF NOT EXISTS terminals (
id bigserial PRIMARY KEY,
merchant_id bigint NOT NULL REFERENCES merchants(id),
source_key text NOT NULL,
verify_type text NOT NULL,
UNIQUE (merchant_id, source_key)
);
-- statement-break
CREATE TABLE IF NOT EXISTS payment_sessions (
id bigserial PRIMARY KEY,
import_run_id bigint NOT NULL REFERENCES import_runs(id),
merchant_id bigint NOT NULL REFERENCES merchants(id),
terminal_id bigint NOT NULL REFERENCES terminals(id),
source_key text NOT NULL,
status text NOT NULL,
amount_rial bigint NOT NULL CHECK (amount_rial >= 0),
fee_amount_rial bigint NOT NULL CHECK (fee_amount_rial >= 0),
payer_card_key text,
created_at timestamptz NOT NULL,
expires_at timestamptz,
verified_at timestamptz,
settled_at timestamptz,
UNIQUE (merchant_id, source_key)
);
-- statement-break
CREATE TABLE IF NOT EXISTS payment_attempts (
id bigserial PRIMARY KEY,
session_id bigint NOT NULL REFERENCES payment_sessions(id) ON DELETE CASCADE,
sequence integer NOT NULL CHECK (sequence > 0),
status text NOT NULL,
created_at timestamptz,
init_time_ms integer,
verify_time_ms integer,
UNIQUE (session_id, sequence)
);
-- statement-break
CREATE INDEX IF NOT EXISTS payment_sessions_merchant_created_idx ON payment_sessions (merchant_id, created_at DESC, id DESC);
-- statement-break
CREATE INDEX IF NOT EXISTS payment_sessions_status_idx ON payment_sessions (merchant_id, status, created_at DESC);
-- statement-break
CREATE INDEX IF NOT EXISTS payment_attempts_session_idx ON payment_attempts (session_id, sequence);
-- statement-break
CREATE TABLE IF NOT EXISTS merchant_users (
id bigserial PRIMARY KEY,
merchant_id bigint NOT NULL REFERENCES merchants(id),
username text NOT NULL UNIQUE,
password_hash text NOT NULL,
is_active boolean NOT NULL DEFAULT true,
created_at timestamptz NOT NULL DEFAULT now()
);
-- statement-break
CREATE TABLE IF NOT EXISTS auth_sessions (
id bigserial PRIMARY KEY,
user_id bigint NOT NULL REFERENCES merchant_users(id),
merchant_id bigint NOT NULL REFERENCES merchants(id),
token_hash bytea NOT NULL UNIQUE,
csrf_token text NOT NULL,
created_at timestamptz NOT NULL,
idle_expires_at timestamptz NOT NULL,
absolute_expires_at timestamptz NOT NULL,
revoked_at timestamptz,
last_seen_at timestamptz NOT NULL
);
-- statement-break
CREATE INDEX IF NOT EXISTS auth_sessions_token_idx ON auth_sessions (token_hash) WHERE revoked_at IS NULL;
-- statement-break
INSERT INTO schema_migrations(version) VALUES (1) ON CONFLICT DO NOTHING;

222
tasks/plan.md Normal file
View File

@@ -0,0 +1,222 @@
# Implementation Plan: ZarinPal Merchant Analytics Backend
- Status: Approved — Phase 4 implementation authorized 2026-08-20
- Capabilities: Data foundation, demo identity, analytics engine, and Merchant API
- Approved contracts: physical schema, import workflow/validation, Identity, Analytics, metric catalogue `0.4.0`, API, and OpenAPI `1.0.0`
- Task list: `tasks/todo.md`
- Last updated: 2026-08-20
## Overview
Build the complete Go capability-oriented modular monolith needed by the dashboard: reviewed PostgreSQL migrations and normalized entities, the guarded one-time importer, demo merchant identity, deterministic analytics queries, Redis/Asynq orchestration, and the approved `/api/v1` REST/SSE/detail/CSV adapter. Deployment configuration remains owned by the deployment repository.
## Preconditions and boundaries
- The approved Identity, Data Foundation, physical schema, importer, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative.
- Architecture and dependency injection are already approved: capability-oriented modular monolith with manual constructor wiring in thin `cmd/` composition roots.
- The user approved this plan and task list and authorized implementation on 2026-08-20.
- Implementation proceeds through `tasks/todo.md` in dependency order, starting with Task F1.
- The API work depends on implementation-ready provider interfaces from Identity and Analytics. Internal provider signatures may be refined without changing the approved HTTP contract.
- Any OpenAPI generator, validator, router, rate-limiter, or CSV dependency must be version-pinned, checked against official documentation, security-reviewed, and approved before addition.
## Architecture decisions
- `docs/openapi.yaml` remains the public interface source of truth. Generated artifacts are reproducible outputs, never a competing hand-written contract.
- HTTP handlers live in the Merchant API capability and depend on narrow consumer-owned Identity/Analytics interfaces.
- Shared HTTP middleware owns request ids, request-size limits, safe Problem Details serialization, authentication, CSRF/origin enforcement, and no-store/security headers.
- Merchant identity is never accepted from a path, query, header, request body, cursor, or event id. It is resolved from the current server-side session on every protected request.
- JSON detail and CSV export share one canonical filter/authorization/projection policy so export cannot widen the authorized view.
- SSE reauthorizes connection and replay, uses retained opaque event ids, and resets to a current snapshot when replay is unavailable.
- Audit and observability record safe metadata and correlation ids, not payment rows, credentials, tokens, raw cursors, or peer data.
## Dependency graph
```text
Go scaffold/tooling
|
v
Ent model + reviewed Atlas migrations
|
+--> pgx staging importer --> validation/promotion/recovery
| |
| v
| sole immutable dataset snapshot
|
+--> demo identity/session service
|
+--> calendar + metric query kernel
|
v
Redis/Asynq run orchestration
|
v
Approved OpenAPI --> HTTP boundary --> auth/period/run/SSE/detail/CSV
|
v
audit + full verification
```
## Ordered task index
### Foundation phase A: repository and schema
- [ ] Task F1: Establish the Go workspace, command roots, configuration, and quality toolchain.
- [ ] Task F2: Implement the normalized Ent model and reviewed Atlas migration baseline.
### Checkpoint F-A: Schema foundation
- [ ] Clean migrations create the approved physical schema and ERD relationships without triggers or revision tables.
- [ ] Build, unit-test, lint, vulnerability, and migration commands are reproducible.
### Foundation phase B: one-time dataset bootstrap
- [ ] Task F3: Implement pinned archive preflight and bounded pgx staging.
- [ ] Task F4: Implement set-based validation, atomic promotion, redacted reports, and recovery.
- [ ] Task F5: Implement deterministic synthetic merchant profiles and secure demo-account seeding.
### Checkpoint F-B: Trusted dataset
- [ ] The full pinned archive reconciles to approved counts and the second identical run is a no-op.
- [ ] Fatal input, interruption, concurrency, and sensitive-data leak tests pass.
### Foundation phase C: identity and analytics providers
- [ ] Task F6: Implement password authentication and revocable server-side merchant sessions.
- [ ] Task F7: Implement calendar periods and the deterministic metric query kernel.
- [ ] Task F8: Implement Redis/Asynq analytics runs, section cache, retries, and event history.
- [ ] Task F9: Implement tenant-scoped detail queries and shared export projection.
- [ ] Task F10: Stabilize provider interfaces and complete backend-provider integration checks.
### Checkpoint F-C: Provider readiness
- [ ] Identity, periods, metrics, peer suppression, runs, events, detail, and export providers pass PostgreSQL/Redis integration tests.
- [ ] Provider contracts expose the immutable dataset snapshot and no client-supplied merchant authority.
### API phase 1: Contract and HTTP boundary
- [ ] Task 1: Establish the OpenAPI toolchain and conformance harness.
- [ ] Task 2: Build the shared HTTP boundary and protected-request middleware.
### Checkpoint 1: Boundary readiness
- [ ] OpenAPI validation/generation is reproducible from a clean checkout.
- [ ] Shared Problem Details, request limits, request ids, cookie authentication, and CSRF/origin behavior pass focused tests.
- [ ] Human review confirms no application handler can obtain tenant authority from client input.
### API phase 2: Core vertical slices
- [ ] Task 3: Implement login, current-session, and logout endpoints.
- [ ] Task 4: Implement approved reporting-period discovery.
- [ ] Task 5: Implement analytics-run start and snapshot retrieval.
### Checkpoint 2: Core API flow
- [ ] A merchant can authenticate, obtain an approved period, start one deduplicated run, and read its snapshot.
- [ ] Cross-tenant ids are indistinguishable from missing ids.
- [ ] OpenAPI examples and focused integration tests pass.
### API phase 3: Partial delivery and recovery
- [ ] Task 6: Implement authenticated SSE delivery and recovery.
- [ ] Task 7: Implement exhausted-section manual retry.
### Checkpoint 3: Asynchronous behavior
- [ ] Independent section completion, heartbeat, replay, snapshot reset, revocation, and sibling-failure isolation pass integration tests.
- [ ] Existing start/retry/SSE limits reject before queue or stream work.
### API phase 4: Evidence and export
- [ ] Task 8: Implement cursor-paginated session detail views.
- [ ] Task 9: Implement safeguarded Persian CSV export.
### Checkpoint 4: Reconciliation and data boundary
- [ ] Each detail view reconciles with its approved aggregate.
- [ ] JSON and CSV use identical tenant/filter predicates and approved field projections.
- [ ] Payer tokens, credentials, peer rows, and internal keys are absent from responses, exports, logs, and traces.
### API phase 5: Operational hardening
- [ ] Task 10: Publish the stable API problem-type catalogue.
- [ ] Task 11: Add API audit and observability instrumentation.
- [ ] Task 12: Complete contract, security, race, and performance verification.
### Checkpoint 5: Implementation-ready completion
- [ ] All OpenAPI operations and documented status codes have passing contract tests.
- [ ] `go test`, race detection, `go vet`, `golangci-lint`, and `govulncheck` pass using repository-approved commands.
- [ ] Runtime checks cover tenant isolation, session revocation, SSE recovery, cursor integrity, export caps, formula neutralization, and sensitive-field absence.
- [ ] Documentation matches runtime behavior and the full project Definition of Done is satisfied.
- [ ] Human review approves completion before merge or deployment.
## Verification strategy
### Per task
- Write tests that fail before the behavior exists and pass after it is implemented.
- Run focused package tests for the changed slice.
- For schema/import slices, compare migrations to the approved ERD, exercise rollback/recovery, and reconcile pinned archive counts without committing source data.
- For metric slices, compare results to an independent SQL oracle across Tehran/Jalali boundaries, missing values, status semantics, peers, and large/skewed merchants.
- For identity slices, exercise enumeration resistance, two-session enforcement, expiry/revocation, CSRF, throttling, tenant isolation, and sensitive-value absence.
- For Redis/Asynq slices, exercise single-flight, independent partial completion, bounded retries, replay, rate limits, and safe cold recomputation after Redis loss.
- Validate `docs/openapi.yaml` and regenerate/check generated artifacts without diff.
- Run `go test ./...` and `go vet ./...` once the Go scaffold exists.
- Inspect the staged diff and scan it for secrets and forbidden data fields before every commit.
### At checkpoints
- Run `go test -race ./...`, `golangci-lint run`, and `govulncheck ./...` after the required tools are pinned by the approved scaffold/tooling task.
- Exercise the actual HTTP server against PostgreSQL and Redis test dependencies for authentication, tenant isolation, analytics lifecycle, SSE, pagination, and export behavior.
- Run OpenAPI request/response conformance tests for all examples, errors, media types, headers, and security requirements.
- Verify query plans and bounded resource behavior for detail/export paths with representative merchant volumes.
## Parallelization
- F2 follows F1. After F2, importer preflight/staging (F3), metric-query work (F7), and the non-credential portion of synthetic profile design (F5) may proceed independently.
- F4 follows F3. Credential seeding completes only after F4, and session work (F6) follows F5.
- Run orchestration (F8) and detail/export providers (F9) may proceed in parallel after their identity and metric dependencies are stable; F10 integrates them sequentially.
- API contract tooling (Task 1) may proceed alongside provider work after F1, but protected endpoints wait for their named F-provider dependencies.
- Tasks 3 and 4 may proceed in parallel only after Tasks 1–2 stabilize shared HTTP types and middleware.
- Task 6 and the read-only parts of Task 8 may proceed in parallel after Task 5 and the Analytics provider interfaces stabilize.
- Task 9 is sequential after Task 8 because CSV must reuse its authorization, filters, and projection.
- Tasks 10–11 may proceed alongside later feature slices once stable error codes and audit events exist.
- Task 12 is sequential after all feature slices and instrumentation are complete.
## Risks and mitigations
| Risk | Impact | Mitigation |
|---|---|---|
| Import accepts corrupt/oversized data or partially promotes | Critical | Pinned preflight, bounded streaming, set-based validation, atomic promotion, exact reconciliation, fault-injection and recovery tests |
| Physical schema or metrics drift from approved semantics | Critical | ERD/migration comparison, independent SQL oracle, boundary fixtures, catalogue-version evidence |
| Demo authentication consumes effort or becomes production identity | Medium | Keep the bounded demo-only session scope and preserve the explicit future ZarinPal SSO replacement boundary |
| Generated types drift from OpenAPI | High | Reproducible pinned generation plus clean-diff and contract-test gates |
| Tenant scope leaks through ids/cursors/replay | Critical | Session-derived merchant context, ownership checks in providers, identical not-found policy, adversarial integration tests |
| SSE behavior differs across proxy/browser boundaries | High | Test event framing, heartbeat, buffering headers, replay, reset, disconnect, and revocation through the deployed-style proxy path |
| CSV widens detail access or leaks formulas/sensitive data | Critical | Reuse one canonical detail query/projection, preflight limits, formula neutralization, allowlist tests, abort-aware audit result |
| Retry/idempotency races duplicate jobs | High | Atomic intent claim, canonical request hash, current-run deduplication, concurrency tests |
| Large merchant queries exhaust API resources | High | Cursor pagination, fixed filters, timeouts, export preflight caps, measured plans, bounded streaming |
| Error details expose internals | Medium | Central RFC 9457 mapper with stable allowlisted codes and negative leak tests |
| Pre-1.0 queue behavior changes | High | Preserve approved Asynq `v0.26.0` pin and run retry/deduplication/recovery integration tests before any upgrade |
## Implementation gates, not product questions
- Select and approve exact pinned OpenAPI generation/validation tooling before adding dependencies.
- Confirm repository-wide build/test/lint command wrappers when the authorized Go scaffold is created.
- Confirm the hosting mechanism that makes `https://zarinpal.amiirkhl.ir/docs/errors/{error-slug}` dereferenceable; the URI namespace and content responsibility are already approved.
## Product questions
None.
## Approved feature-branch redesign API increment
The user approved the merchant-dashboard redesign on 2026-08-20. The backend increment is additive and follows this dependency order:
1. Pure, tested Jalali boundary helpers and a dataset-bounded catalog of all complete supported period descriptors.
2. Authenticated merchant-terminal listing and optional terminal scope on analytics runs.
3. Bounded overview payment-history points for the selected scope.
4. Transaction listing filters and one merchant-owned transaction detail resource.
5. OpenAPI regeneration, local database smoke, and feature preview on port `8081`.
No migration is required. All terminal and session identifiers are resolved under the merchant from the authenticated server session.

566
tasks/todo.md Normal file
View File

@@ -0,0 +1,566 @@
# ZarinPal Merchant Analytics Backend Task List
- Status: Approved — Phase 4 implementation authorized 2026-08-20
- Plan: `tasks/plan.md`
- Contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0`
Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. Foundation tasks use `F` identifiers; the previously approved API tasks retain their stable numeric identifiers.
## Task F1: Establish the Go workspace and quality toolchain
**Description:** Create the thin command-root/modular-monolith skeleton, validated configuration boundary, structured logging baseline, and pinned repository quality commands without implementing product behavior.
**Acceptance criteria:**
- [ ] `cmd/api`, `cmd/worker`, `cmd/importer`, and migration/admin composition roots are thin and business packages live under `internal/`.
- [ ] Manual constructor wiring, context propagation, validated environment configuration, and safe structured logs follow approved ADRs/AGENTS.
- [ ] Pinned build/test/race/vet/lint/vulnerability/generation commands are documented and reproducible.
**Verification:** Run every scaffold quality command from a clean checkout; confirm no service starts implicitly and dependency/checksum diffs are approved.
**Dependencies:** Explicit implementation authorization.
**Files likely touched:** `go.mod`, `go.sum`, `cmd/`, `internal/platform/`, quality-tool configuration.
**Estimated scope:** Medium (split command roots if more than five files)
## Task F2: Implement the normalized Ent model and migration baseline
**Description:** Translate the approved physical schema into Ent entities/relationships and reviewed Atlas SQL migrations, including identity, import, payment, calendar, and audit tables without cache-revision triggers.
**Acceptance criteria:**
- [ ] Tables, FKs, checks, unique/partial indexes, nullability, restrictive deletion, and 30/90-day audit retention match the physical schema.
- [ ] `payment_sessions.import_run_id` is required; `NoAttempt` never becomes an attempt; terminal verification method and session-grain money are stored once.
- [ ] Clean creation and supported upgrade migrations are versioned, reviewed, and never run from API startup.
**Verification:** Run zero-to-head/upgrade migration tests, schema introspection/ERD comparison, constraint negatives, and representative query-plan checks.
**Dependencies:** F1.
**Files likely touched:** `internal/data/schema/`, `ent/schema/`, `migrations/`, migration tests, Atlas configuration.
**Estimated scope:** Medium slices by schema group
## Task F3: Implement pinned archive preflight and bounded staging
**Description:** Build the one-shot importer boundary that validates the fixed file/checksum/schema/resource caps and streams strict CSV/gzip records through pgx `COPY` into run-scoped staging.
**Acceptance criteria:**
- [ ] Fixed filename, SHA-256, 22-column schema, UTF-8/gzip integrity, byte/row/record caps, and regular read-only path checks fail closed.
- [ ] Parsing is bounded-memory, independent of file/session order, and records safe row numbers without logging source values.
- [ ] One PostgreSQL advisory lock prevents concurrent validate/import/recovery commands.
**Verification:** Unit/fuzz tests cover malformed CSV/UTF-8/gzip, overflow, traversal/symlink, expansion, cancellation, caps, and redaction; staging load stays within measured memory.
**Dependencies:** F1–F2.
**Files likely touched:** `cmd/importer/`, `internal/importing/preflight/`, `internal/importing/staging/`, focused tests.
**Estimated scope:** Medium, split preflight and staging commits
## Task F4: Implement validation, atomic promotion, reports, and recovery
**Description:** Add set-based relationship validation, warning/fatal classification, exact reconciliation, transactional normalization, redacted reports, staging cleanup, no-op detection, and restart-from-source recovery.
**Acceptance criteria:**
- [ ] Any fatal issue promotes zero normalized facts; documented timestamp/sequence/null anomalies remain unchanged warnings.
- [ ] Pinned counts and per-merchant/status reconciliations pass before the sole import run becomes completed.
- [ ] Success/failure removes raw staging; interrupted recovery cleans only its run scope and restarts from the immutable archive.
**Verification:** Full-archive rehearsal plus relationship/rollback/fault-injection/concurrency/no-op/recovery/leak tests; JSON and Markdown reports reconcile.
**Dependencies:** F3.
**Files likely touched:** `internal/importing/validation/`, `internal/importing/promotion/`, `internal/importing/report/`, recovery command/tests.
**Estimated scope:** Medium slices; never one XL commit
## Task F5: Implement synthetic merchant profiles and demo-account seeding
**Description:** Generate deterministic non-sensitive merchant display profiles and exactly one initial demo user per merchant through a separate secure administrative flow.
**Acceptance criteria:**
- [ ] Display names are deterministic, clearly synthetic in storage, and stable across identical imports.
- [ ] Usernames are unique and credentials use approved Argon2id hashing; plaintext is emitted once only to an explicit protected output.
- [ ] Re-running the seed is idempotent and never derives passwords from merchant/source keys.
**Verification:** Determinism/idempotency/hash/permission/redaction tests and two-merchant credential smoke fixture.
**Dependencies:** F2 and completed F4 provider behavior.
**Files likely touched:** `cmd/admin/`, `internal/identity/seeding/`, profile generator/tests, operator output adapter.
**Estimated scope:** Medium (3–5 files)
## Task F6: Implement password authentication and server-side sessions
**Description:** Implement demo credential verification, generic throttled outcomes, two-active-session enforcement, opaque token digests, CSRF binding, expiry/activity coalescing, logout/revocation, and authentication audit retention.
**Acceptance criteria:**
- [ ] Authentication never reveals account existence and returns tenant context only from a valid server-side session.
- [ ] Cookie/token/CSRF, idle/absolute expiry, revocation, disabled account, and two-session behavior match Identity spec.
- [ ] Raw credentials/tokens/IP/user-agent values are absent from storage, logs, and audit evidence.
**Verification:** Table/race/integration tests cover enumeration, throttles, fixation, rotation, expiry, revocation, concurrency, and 30-day cleanup.
**Dependencies:** F2 and F5.
**Files likely touched:** `internal/identity/`, session repository, throttling/audit adapters, focused tests.
**Estimated scope:** Medium slices
## Task F7: Implement reporting periods and deterministic metric queries
**Description:** Build calendar-period discovery and SQL/Ent query services for catalogue `0.4.0`, preserving session grain, missingness, Saturday–Friday/Jalali/season boundaries, and peer suppression/equal weighting.
**Acceptance criteria:**
- [ ] Every catalogue metric reconciles for day/week/month/season and emits unit, denominator/sample/coverage, dataset snapshot, metric version, and calculation time.
- [ ] `Paid` plus `Verified`, verification completion, retry recovery, timing percentiles, fees, and `NoAttempt` semantics match approved formulas.
- [ ] Peer outputs exclude the active merchant, weight merchants equally, invert lower-is-better percentiles, and suppress below both thresholds.
**Verification:** Independent SQL oracle tests, boundary fixtures, large/skewed merchant fixtures, null/zero/unknown-status tests, and measured query plans.
**Dependencies:** F2 and trusted F4 dataset fixtures.
**Files likely touched:** `internal/analytics/periods/`, `internal/analytics/metrics/`, peer queries, fixtures/tests.
**Estimated scope:** Medium slices by section
## Task F8: Implement Redis/Asynq analytics run orchestration
**Description:** Implement idempotent four-section runs, dataset-versioned 30-day cache keys, Asynq `v0.26.0` jobs, bounded concurrency, retries, rate limits, event history, and sibling-failure isolation.
**Acceptance criteria:**
- [ ] Equivalent starts single-flight; all sections begin in bounded parallel and publish ready results atomically/independently.
- [ ] Attempts time out at 60 seconds and only transient failures receive two exponential-backoff/jitter retries.
- [ ] Redis loss yields safe cold recomputation; one-hour events/15-second heartbeat and independent user/merchant limits match specs.
**Verification:** Queue/cache integration, race, failure, retry timing, Redis-loss, deduplication, replay, rate-limit, and sensitive-payload tests.
**Dependencies:** F1, F6, and F7.
**Files likely touched:** `internal/analytics/runs/`, `internal/analytics/jobs/`, Redis adapters, worker root/tests.
**Estimated scope:** Medium slices
## Task F9: Implement tenant-scoped detail and export providers
**Description:** Build the shared authorized query/projection for unverified-paid, failed, and retried details, deterministic keyset pagination, export preflight/count/stream inputs, and operator audit metadata.
**Acceptance criteria:**
- [ ] Every query requires session-derived merchant scope and approved period/filter allowlists.
- [ ] Detail/export projections reconcile to metrics and exclude payer tokens, peer rows, credentials, and internal keys.
- [ ] Stable ordering, cursor inputs, 100,000-row/50-MiB/60-second export caps, and audit metadata are available to the API adapter.
**Verification:** Tenant/tampering/reconciliation/pagination/snapshot/cap/query-plan/leak tests against PostgreSQL.
**Dependencies:** F2, F6, and F7.
**Files likely touched:** `internal/analytics/details/`, cursor/projection package, export preflight, audit/tests.
**Estimated scope:** Medium slices
## Task F10: Stabilize provider interfaces and backend integration
**Description:** Wire Identity, period, analytics-run/event, detail/export, audit, PostgreSQL, and Redis providers behind narrow interfaces consumed by the Merchant API.
**Acceptance criteria:**
- [ ] Composition roots wire explicit constructors with bounded pools/timeouts and graceful lifecycle.
- [ ] Provider contracts match OpenAPI semantics without importing HTTP types into domain packages.
- [ ] A non-HTTP integration harness completes seed/login/period/run/partial-result/detail/export-provider flows for two isolated merchants.
**Verification:** Full provider integration, race, shutdown, dependency-loss, tenant-boundary, and safe-log tests.
**Dependencies:** F4–F9.
**Files likely touched:** provider interfaces/adapters, `cmd/api/`, `cmd/worker/`, integration tests.
**Estimated scope:** Medium (split wiring and integration)
## Checkpoint F: Backend provider readiness
- [ ] Tasks F1–F10 and every foundation acceptance criterion pass.
- [ ] Full pinned import, two-merchant identity, metric reconciliation, Redis-loss, and tenant isolation evidence is reviewed.
- [ ] Human approves provider readiness before completing HTTP feature slices.
## Task 1: Establish the OpenAPI toolchain and conformance harness
**Description:** Select, approve, and pin the Go OpenAPI generation/validation tooling; make contract validation and generated-artifact drift checks reproducible from a clean checkout.
**Acceptance criteria:**
- [ ] One approved pinned toolchain validates OpenAPI 3.1 and produces only the intended server/types artifacts.
- [ ] Generated artifacts contain no hand-edited business logic and regenerate with a clean Git diff.
- [ ] Stable `operationId` values, schemas, media types, and security requirements are covered by a contract test harness.
**Verification:**
- [ ] Run the documented OpenAPI validate/generate/check commands from a clean checkout.
- [ ] Run focused generator/conformance tests and confirm a deliberate contract mismatch fails.
- [ ] Review generated and lockfile/checksum diffs; approve any new dependency before installation.
**Dependencies:** Explicit implementation authorization and F1.
**Files likely touched:**
- `go.mod`
- `go.sum`
- `internal/api/contract/`
- `internal/api/contract_test.go`
- approved command/tool configuration
**Estimated scope:** Medium (3–5 files/artifact groups)
## Task 2: Build the shared HTTP boundary and protected-request middleware
**Description:** Create the API server boundary for `/api/v1`, central Problem Details mapping, request ids, 16 KiB body/8 KiB target limits, no-store/security headers, session resolution, and CSRF/origin enforcement.
**Acceptance criteria:**
- [ ] All protected handlers receive merchant/user context only from a verified server-side session.
- [ ] Boundary failures use the approved RFC 9457 schema/code/status mapping without internal details.
- [ ] Request limits, unknown fields/parameters, CSRF/origin checks, and no-store/security headers behave as specified.
**Verification:**
- [ ] Focused middleware tests cover missing/expired/revoked sessions, malformed input, size limits, request ids, CSRF, and origin/fetch metadata.
- [ ] Cross-tenant client inputs cannot create or replace merchant context.
- [ ] `go test ./...` and `go vet ./...` pass.
**Dependencies:** Task 1 and F6 provider interface.
**Files likely touched:**
- `internal/api/http/server.go`
- `internal/api/http/problem.go`
- `internal/api/http/middleware.go`
- `internal/api/http/middleware_test.go`
**Estimated scope:** Medium (4 files)
## Checkpoint 1: Boundary readiness
- [ ] Tasks 1–2 acceptance criteria pass.
- [ ] OpenAPI generation is reproducible with a clean diff.
- [ ] Human review confirms the tenant and error boundaries before feature handlers begin.
## Task 3: Implement authentication session endpoints
**Description:** Implement login, current-session, and logout as one vertical slice through the approved Identity provider, including secure cookie and generic-failure behavior.
**Acceptance criteria:**
- [ ] Login creates a revocable session and sets only the approved `__Host-zpa_session` cookie attributes.
- [ ] Current-session JSON matches OpenAPI and exposes a session-bound CSRF token but no authentication token.
- [ ] Logout revokes server state before clearing the cookie; expected credential failures remain indistinguishable and throttled.
**Verification:**
- [ ] Integration tests cover success, enumeration resistance, cookie flags, rotation, expiry, revocation, throttling, CSRF, and dependency failure.
- [ ] OpenAPI conformance tests cover all auth statuses and headers.
- [ ] `go test -race ./...` passes for affected packages.
**Dependencies:** Task 2 and F6.
**Files likely touched:**
- `internal/api/http/auth_handler.go`
- `internal/api/http/auth_handler_test.go`
- `internal/api/identity.go`
- `cmd/api/main.go`
**Estimated scope:** Medium (4 files)
## Task 4: Implement approved reporting-period discovery
**Description:** Return only allowlisted day/week/Jalali-month/Persian-season periods with exact UTC/Tehran boundaries, ready Persian labels, and the approved default.
**Acceptance criteria:**
- [ ] Period responses match OpenAPI and use backend-calculated half-open boundaries.
- [ ] Iranian weeks start Saturday; default selection is the latest complete Jalali month containing source data.
- [ ] Unknown parameters/types are rejected and arbitrary date ranges are impossible.
**Verification:**
- [ ] Table tests cover all period types, month/year boundaries, Saturday–Friday weeks, empty source ranges, and timezone independence.
- [ ] Integration tests reconcile returned ids/boundaries with `calendar_dates`.
- [ ] OpenAPI examples and `go test ./...` pass.
**Dependencies:** Task 2 and F7.
**Files likely touched:**
- `internal/api/http/period_handler.go`
- `internal/api/http/period_handler_test.go`
- `internal/api/periods.go`
**Estimated scope:** Medium (3 files)
## Task 5: Implement analytics-run start and snapshot retrieval
**Description:** Implement idempotent run creation/current-run resolution and tenant-scoped snapshot retrieval for all four sections.
**Acceptance criteria:**
- [ ] Same intent/key replays safely, key/body conflicts fail, and equivalent concurrent starts resolve one current run.
- [ ] Run snapshots serialize every section state/result exactly as OpenAPI defines.
- [ ] Missing and cross-tenant run ids return the same `404 RESOURCE_NOT_FOUND` response.
**Verification:**
- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, the immutable dataset snapshot, and no duplicate jobs.
- [ ] Contract tests cover `200`, `202`, `401`, `403`, `404`, `422`, `429`, and dependency failures.
- [ ] Integration tests assert that rejected/limited requests enqueue no work.
**Dependencies:** Tasks 1–2 and F8.
**Files likely touched:**
- `internal/api/http/run_handler.go`
- `internal/api/http/run_handler_test.go`
- `internal/api/analytics.go`
- `internal/api/idempotency.go`
**Estimated scope:** Medium (4 files)
## Checkpoint 2: Core API flow
- [ ] Tasks 3–5 acceptance criteria pass.
- [ ] Authenticated merchant flow works through the actual HTTP server and test dependencies.
- [ ] Cross-tenant and idempotency adversarial tests pass.
- [ ] Human review approves the core flow before streaming/detail work.
## Task 6: Implement authenticated SSE delivery and recovery
**Description:** Stream typed run/section events with 15-second heartbeats, one-hour authorized replay, snapshot reset, and session-aware disconnect/reconnect behavior.
**Acceptance criteria:**
- [ ] Event framing, ids, names, and JSON data match the approved SSE schemas and UTF-8 media type.
- [ ] Retained ids replay only later authorized events; unavailable ids reset with `run.snapshot` and never disclose another tenant.
- [ ] Session expiry/revocation, connection limits, and completed-run closure behave as specified.
**Verification:**
- [ ] Integration tests cover ordered partial readiness, failure isolation, suppression, heartbeat timing, replay, reset, disconnect, revocation, and user/merchant connection limits.
- [ ] Proxy-style tests assert no buffering/compression and required stream headers.
- [ ] Race tests cover concurrent publish/replay/disconnect paths.
**Dependencies:** Task 5 and F8.
**Files likely touched:**
- `internal/api/http/sse_handler.go`
- `internal/api/http/sse_handler_test.go`
- `internal/api/events.go`
- `internal/api/events_test.go`
**Estimated scope:** Medium (4 files)
## Task 7: Implement exhausted-section manual retry
**Description:** Add the idempotent retry-attempt subresource for one exhausted failed section without disturbing ready siblings.
**Acceptance criteria:**
- [ ] Only an exhausted, retryable failed section for the current run/dataset snapshot is accepted.
- [ ] Retry idempotency and independent user/merchant throttles prevent duplicate work.
- [ ] Ready siblings remain unchanged and stale/cross-tenant requests fail safely.
**Verification:**
- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale dataset snapshots, throttling, cross-tenant ids, and no sibling restart.
- [ ] Rejected retries enqueue no work.
- [ ] OpenAPI conformance and `go test -race ./...` pass.
**Dependencies:** Tasks 5–6.
**Files likely touched:**
- `internal/api/http/retry_handler.go`
- `internal/api/http/retry_handler_test.go`
- `internal/api/retry.go`
**Estimated scope:** Medium (3 files)
## Checkpoint 3: Asynchronous behavior
- [ ] Tasks 6–7 acceptance criteria pass.
- [ ] SSE and retry runtime checks pass through the actual queue/cache test boundary.
- [ ] Human review confirms recovery and rate-limit behavior.
## Task 8: Implement cursor-paginated session detail views
**Description:** Implement the three approved detail views with deterministic ordering, opaque bound cursors, default 25/maximum 100 rows, and a strict field allowlist.
**Acceptance criteria:**
- [ ] `UNVERIFIED_PAID`, `FAILED`, and `RETRIED` filters match catalogue definitions and reconcile to aggregates.
- [ ] Cursors bind tenant, canonical filters, ordering, expiry, and last row; invalid reuse returns generic `INVALID_CURSOR`.
- [ ] Detail rows expose only approved fields and the tenant-scoped opaque terminal reference.
**Verification:**
- [ ] Tests cover page boundaries, duplicate timestamps, cursor tampering/expiry/filter change/cross-tenant reuse, limits, and empty results.
- [ ] Integration queries reconcile representative merchants and approved periods.
- [ ] Sensitive-field absence and measured query-plan checks pass.
**Dependencies:** Tasks 2 and 4 plus F9.
**Files likely touched:**
- `internal/api/http/detail_handler.go`
- `internal/api/http/detail_handler_test.go`
- `internal/api/details.go`
- `internal/api/cursor.go`
- `internal/api/cursor_test.go`
**Estimated scope:** Medium (5 files)
## Task 9: Implement safeguarded Persian CSV export
**Description:** Stream the same authorized detail predicate/projection as Persian CSV with BOM, Persian spaced headers/statuses, machine-safe numeric/ISO values, formula neutralization, and approved caps.
**Acceptance criteria:**
- [ ] CSV and JSON share one canonical tenant/filter/projection policy; CSV adds no unavailable field.
- [ ] Output matches the approved encoding, headers, labels, quoting, disposition, no-store, and `nosniff` contract.
- [ ] Preflight rejects over 100,000 rows/50 MiB, generation ends within 60 seconds, throttling applies, and interrupted streams are audited as incomplete.
**Verification:**
- [ ] Golden tests cover Persian output, BOM, CRLF, quotes/newlines, all status mappings, and stable headers with spaces.
- [ ] Security tests cover `=`, `+`, `-`, `@`, tab, CR, LF formula prefixes and forbidden-field absence.
- [ ] Integration tests cover filter parity, snapshot consistency, caps, deadline, disconnect, throttle, and content headers.
**Dependencies:** Task 8.
**Files likely touched:**
- `internal/api/http/export_handler.go`
- `internal/api/http/export_handler_test.go`
- `internal/api/export.go`
- `internal/api/export_test.go`
**Estimated scope:** Medium (4 files)
## Checkpoint 4: Reconciliation and data boundary
- [ ] Tasks 8–9 acceptance criteria pass.
- [ ] Detail/CSV reconciliation and sensitive-data negative tests pass.
- [ ] Export resource limits hold under representative merchant volume.
- [ ] Human review approves the merchant evidence boundary.
## Task 10: Publish the stable API problem-type catalogue
**Description:** Document every stable application error code and make each approved `https://zarinpal.amiirkhl.ir/docs/errors/{error-slug}` URI resolve to safe human-readable guidance.
**Acceptance criteria:**
- [ ] Every OpenAPI problem code maps to one stable slug, title, status, meaning, safe resolution guidance, and retry rule.
- [ ] Published pages expose no internal topology, SQL, stack, secret, tenant existence, or security-control detail useful for abuse.
- [ ] Contract tests fail when a returned problem type/code lacks catalogue documentation.
**Verification:**
- [ ] Link checks confirm every production type URI resolves successfully.
- [ ] OpenAPI and catalogue mapping tests pass with no duplicate slug/code.
- [ ] Security review approves public wording.
**Dependencies:** Tasks 2–9 for the final error-code set; hosting mechanism approval.
**Files likely touched:**
- `docs/ERROR-CATALOGUE.md`
- error-page source/template location approved by the hosting plan
- `internal/api/http/problem_catalogue_test.go`
**Estimated scope:** Medium (2–3 files/artifact groups)
## Task 11: Add API audit and observability instrumentation
**Description:** Instrument the approved audit events, metrics, logs, and traces with bounded-cardinality safe metadata and 90-day analytics/export-audit retention.
**Acceptance criteria:**
- [ ] Required auth/run/SSE/detail/export audit outcomes and operational metrics are emitted with correlation ids.
- [ ] Only authorized backend operators can access analytics/export audit records; automatic deletion occurs after 90 days.
- [ ] Telemetry excludes credentials, tokens, raw cursors/filters, payment rows, payer tokens, peer data, and high-cardinality sensitive labels.
**Verification:**
- [ ] Tests assert required event/metric emission for success, rejection, failure, timeout, and interrupted export.
- [ ] Negative leak tests inspect structured logs, traces, and audit records.
- [ ] Retention/access-control tests pass using an accelerated test clock.
**Dependencies:** Tasks 3–10 and the approved audit/observability adapters.
**Files likely touched:**
- `internal/api/observability.go`
- `internal/api/audit.go`
- `internal/api/observability_test.go`
- `internal/api/audit_test.go`
**Estimated scope:** Medium (4 files)
## Task 12: Complete contract, security, race, and performance verification
**Description:** Run and close the full implementation Definition of Done across all API operations and critical failure/abuse paths.
**Acceptance criteria:**
- [ ] Every OpenAPI operation/status/media type/header/security requirement has automated contract coverage.
- [ ] Tenant isolation, session/CSRF, idempotency, SSE, cursor, export, audit, and sensitive-field abuse suites pass.
- [ ] Representative detail/export query plans and resource bounds meet approved behavior with no unreviewed exception.
**Verification:**
- [ ] Run `go test ./...`, `go test -race ./...`, `go vet ./...`, `golangci-lint run`, and `govulncheck ./...`.
- [ ] Run reproducible OpenAPI validation/generation and confirm a clean Git diff.
- [ ] Exercise critical merchant flows end to end through the deployed-style proxy/test environment.
**Dependencies:** Tasks 1–11.
**Files likely touched:**
- API contract/integration test packages
- approved test fixtures containing no sensitive source data
- API runbook/documentation corrections if verification finds drift
**Estimated scope:** Medium (3–5 files/artifact groups)
## Checkpoint 5: Complete
- [ ] Tasks 1–12 and every checkpoint pass.
- [ ] Full project Definition of Done is satisfied.
- [ ] No unrelated frontend, deployment, physical-schema, ERD, or raw-data changes are included.
- [ ] Human review approves completion before merge or deployment.
## Approved redesign API increment — 2026-08-20
- [ ] Task 13: Return every complete dataset-backed `DAY`, `WEEK`, `JALALI_MONTH`, and `PERSIAN_SEASON` period with the latest complete Jalali month as default.
- Acceptance: pure boundary tests cover Nowruz, month rollover, leap Esfand, Iranian Saturday–Friday weeks, and dataset boundaries.
- Verify: focused analytics tests and `go test ./...`.
- [ ] Task 14: Add authenticated merchant terminal discovery and terminal-scoped analytics runs.
- Acceptance: unknown or other-merchant terminals are rejected without changing merchant scope; an empty terminal means all merchant terminals.
- Verify: service/handler tests plus local authenticated smoke.
- [ ] Task 15: Add bounded daily payment-history points to overview data.
- Acceptance: each point contains date, bank-paid amount, bank-paid sessions, and total sessions for the selected period/scope.
- Verify: query reconciliation against aggregate metrics.
- [ ] Task 16: Add the `ALL` transaction view, status and amount filters, and a merchant-owned transaction detail resource.
- Acceptance: filters are allowlisted/parameterized, pagination happens after filters, and attempts are ordered without payer tokens.
- Verify: focused tests, local drill-down smoke, and safe `404` check.
- [ ] Task 17: Update OpenAPI and run the complete backend gate.
- Acceptance: generated frontend types compile and the feature API runs beside the current API on port `8081`.
- Verify: `go test ./...`, `go vet ./...`, build, and authenticated HTTP smoke.