zarinpal-analytics-backend/docs/DATA-FOUNDATION-SPEC.md

# Data Foundation Capability Spec

- Status: Approved specification
- Implementation authorization: Not granted
- Capability: Data foundation
- Last updated: 2026-08-20

## Objective

Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project.

## In scope

- Streamed `.csv.gz` ingestion from a local, Git-ignored input path
- One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
- PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
- Development seed/reporting workflow and migration/rollback contract

## Out of scope

- Dashboard UI
- Merchant-facing API endpoints
- AI recommendations
- Production backup mechanics (owned by platform operations)
- Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path
- Treating the challenge archive or local database as version-controlled source

## Proposed logical model (technology-neutral)

```mermaid
erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      bigint fee_amount_rial
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      timestamptz expires_at
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }
    CALENDAR_DATE {
      date date PK
      int jalali_year
      int jalali_month
      int jalali_day
      text persian_season
      int iso_weekday
      date reporting_week_start
      date reporting_week_end
    }
```

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, cache/invalidation state, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

## Import contract

This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline.

1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`.
2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
3. Validate headers and supported schema version before database writes.
4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.

## Production write and lazy analytics-cache contract

- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
- Use Asynq pinned to `v0.26.0` as the Redis-backed task queue. Asynq is pre-`v1.0.0`; never use a floating version, and review API/behavior compatibility before upgrading the pin.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL.
- A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
- Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
- Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings.
- Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
- Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available.
- Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return `429` with `Retry-After` and do not enqueue work.
- SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream.
- Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections.
- Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.

## Synthetic merchant/account rules

- Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
- Generate stable display names from a deterministic seed so re-imports do not rename merchants.
- Mark every generated profile field as synthetic in storage.
- Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
- Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

## Acceptance criteria

- A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
- Re-running the same archive is safe and does not duplicate business rows.
- Source row count and accepted/rejected counts reconcile exactly.
- Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
- `(source_session_key, try_seq)` is unique.
- Missing values remain missing rather than being silently replaced with factual-looking defaults.
- The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
- Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
- Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics.
- Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
- Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
- Source `adjusted_fee` is normalized once per session as `fee_amount_rial`; retry attempts never multiply fee totals.
- First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready.
- A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation.
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
- Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
- One failed section does not hide ready sections and does not create a ready cache entry.
- A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately.
- Start, manual-retry, and SSE connection limits return a typed `429` with `Retry-After`; rejected requests enqueue no jobs and cannot affect another merchant.
- SSE emits heartbeats within 15 seconds and supports authorized replay for one hour; replay expiry falls back to the current run snapshot without duplicating work.
- Redis loss produces a cold-cache calculation rather than loss of source data.
- Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes.
- Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval.
- Peer comparisons use only other eligible merchants in the active merchant's category and the same selected reporting period.
- An available peer comparison contains the merchant-weighted cohort median and a direction-aware performance percentile. Each eligible peer contributes one metric value regardless of transaction volume; high-volume merchants cannot dominate the result.
- Peer outputs cover payment success, verification completion, retry recovery, processing-time p50/p95, and paid fee rate, and are suppressed unless at least 10 other merchants and 1,000 combined peer sessions contribute.

## Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003.

Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.

Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.

Resolved: peer benchmarks use other merchants in the active merchant's category for the same selected period, excluding the active merchant. Each peer contributes one independently calculated value to a merchant-weighted median, and the response includes a direction-aware merchant performance percentile. This represents a typical peer despite the dataset's extreme merchant-volume skew. At least 10 other merchants and 1,000 combined peer sessions are required; otherwise the benchmark is suppressed while the merchant's own analytics remain available. See ADR 0006.

Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.

Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011.

Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.

Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `Asia/Tehran`, represented as half-open API/database ranges. See ADR 0012.

Resolved: authoritative ZarinPal lifecycle guidance defines both `Paid` and `Verified` as successful bank payments. `Paid` means funds were deducted but merchant verification is incomplete; `Verified` is finalized. Success/paid-volume/recovery metrics include both while verification metrics keep them distinct. Source `adjusted_fee` is treated as the real merchant fee, normalized once per session, and shown in Rial. See ADR 0013 and `METRIC-CATALOGUE.md`.

No broad architecture approval gates remain for this capability. Physical columns, exact SSE event schemas, retryable-error classification, analytics-section response contracts, observability targets, and migration details still require acceptance in the implementation-ready specs.