# Data Foundation Capability Spec - Status: Approved specification - Implementation authorization: Not granted - Capability: Data foundation - Last updated: 2026-08-20 ## Objective Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project. ## In scope - Streamed `.csv.gz` ingestion from a local, Git-ignored input path - One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher - Import runs, checksums, schema version, row counts, rejection/error reporting - Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts - Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes - PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys - Synthetic merchant display profiles and one demo account per merchant - Idempotent re-runs and explicit import lifecycle - Database constraints/indexes driven by documented access paths - Development seed/reporting workflow and migration/rollback contract ## Out of scope - Dashboard UI - Merchant-facing API endpoints - AI recommendations - Production backup mechanics (owned by platform operations) - Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path - Treating the challenge archive or local database as version-controlled source ## Proposed logical model (technology-neutral) ```mermaid erDiagram MERCHANT_CATEGORY ||--o{ MERCHANT : classifies MERCHANT ||--o{ TERMINAL : owns MERCHANT ||--o{ MERCHANT_USER : authenticates_as MERCHANT ||--o{ PAYMENT_SESSION : receives TERMINAL ||--o{ PAYMENT_SESSION : handles PAYMENT_SESSION ||--o{ PAYMENT_ATTEMPT : contains IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by IMPORT_RUN ||--o{ IMPORT_ISSUE : reports MERCHANT_CATEGORY { bigint id PK text source_category_key UK text title_fa } MERCHANT { bigint id PK text source_merchant_key UK bigint category_id FK text display_name boolean profile_is_synthetic } TERMINAL { bigint id PK text source_terminal_key UK bigint merchant_id FK } MERCHANT_USER { bigint id PK bigint merchant_id FK text username UK text password_hash boolean is_demo timestamptz disabled_at } PAYMENT_SESSION { bigint id PK text source_session_key UK bigint merchant_id FK bigint terminal_id FK bigint amount bigint fee_amount_rial text session_status timestamptz created_at bigint import_run_id FK } PAYMENT_ATTEMPT { bigint id PK bigint payment_session_id FK int try_seq "positive" text try_status text switch_response_code text psp_code text issuer_bank_code text payer_card_key text verify_type int init_time_ms int verify_time_ms timestamptz try_created_at timestamptz verified_at timestamptz settled_at timestamptz expires_at } IMPORT_RUN { bigint id PK text source_sha256 text source_schema_version text importer_revision text status bigint observed_rows timestamptz started_at timestamptz completed_at } IMPORT_ISSUE { bigint id PK bigint import_run_id FK text severity text rule_code bigint affected_rows jsonb evidence } CALENDAR_DATE { date date PK int jalali_year int jalali_month int jalali_day text persian_season int iso_weekday date reporting_week_start date reporting_week_end } ``` This ERD is a logical baseline, not an approved physical schema. In particular, source key types, cache/invalidation state, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved. ## Import contract This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline. 1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`. 2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum. 3. Validate headers and supported schema version before database writes. 4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities. 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. A source `(try_seq = 0, try_status = 'NoAttempt')` record creates the session but no `payment_attempt` row; all normalized attempts require `try_seq >= 1`. 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. 8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs. 9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete. ## Production write and lazy analytics-cache contract - ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema. - This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow. - Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract. - An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. - Use Asynq pinned to `v0.26.0` as the Redis-backed task queue. Asynq is pre-`v1.0.0`; never use a floating version, and review API/behavior compatibility before upgrading the pin. - Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL. - A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish. - Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant. - Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings. - Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers. - Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan. - Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available. - Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return `429` with `Retry-After` and do not enqueue work. - SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream. - Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections. - Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation. - Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details. ## Synthetic merchant/account rules - Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality. - Generate stable display names from a deterministic seed so re-imports do not rename merchants. - Mark every generated profile field as synthetic in storage. - Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path. - Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks. ## Acceptance criteria - A clean database can be migrated and populated from the expected archive without loading the full file into process memory. - Re-running the same archive is safe and does not duplicate business rows. - Source row count and accepted/rejected counts reconcile exactly. - Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships. - Source `(session_key, try_seq)` records are unique. Normalized attempts require `try_seq >= 1` and are unique by `(payment_session_id, try_seq)`; a `NoAttempt` marker produces a session with zero attempts. - Missing values remain missing rather than being silently replaced with factual-looking defaults. - The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics. - Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked. - Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics. - Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. - Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. - Source `adjusted_fee` is normalized once per session as `fee_amount_rial`; retry attempts never multiply fee totals. - First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready. - A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation. - Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically. - Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out. - One failed section does not hide ready sections and does not create a ready cache entry. - A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately. - Start, manual-retry, and SSE connection limits return a typed `429` with `Retry-After`; rejected requests enqueue no jobs and cannot affect another merchant. - SSE emits heartbeats within 15 seconds and supports authorized replay for one hour; replay expiry falls back to the current run snapshot without duplicating work. - Redis loss produces a cold-cache calculation rather than loss of source data. - Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes. - Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval. - Peer comparisons use only other eligible merchants in the active merchant's category and the same selected reporting period. - An available peer comparison contains the merchant-weighted cohort median and a direction-aware performance percentile. Each eligible peer contributes one metric value regardless of transaction volume; high-volume merchants cannot dominate the result. - Peer outputs cover payment success, verification completion, retry recovery, processing-time p50/p95, and paid fee rate, and are suppressed unless at least 10 other merchants and 1,000 combined peer sessions contribute. ## Approval gates / unresolved decisions Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001. Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002. Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003. Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004. Resolved: retain `payer_card_key` unchanged as a restricted pseudonymous card identifier whose equality is authoritative only within one merchant. It may support covered repeat-card analytics but cannot identify a person or link merchants. Do not HMAC it, expose it, log it, export it, or send it to AI providers. See ADR 0005. Resolved: peer benchmarks use other merchants in the active merchant's category for the same selected period, excluding the active merchant. Each peer contributes one independently calculated value to a merchant-weighted median, and the response includes a direction-aware merchant performance percentile. This represents a typical peer despite the dataset's extreme merchant-volume skew. At least 10 other merchants and 1,000 combined peer sessions are required; otherwise the benchmark is suppressed while the merchant's own analytics remain available. See ADR 0006. Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011. Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010. Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `Asia/Tehran`, represented as half-open API/database ranges. See ADR 0012. Resolved: authoritative ZarinPal lifecycle guidance defines both `Paid` and `Verified` as successful bank payments. `Paid` means funds were deducted but merchant verification is incomplete; `Verified` is finalized. Success/paid-volume/recovery metrics include both while verification metrics keep them distinct. Source `adjusted_fee` is treated as the real merchant fee, normalized once per session, and shown in Rial. See ADR 0013 and `METRIC-CATALOGUE.md`. No broad architecture approval gates remain for this capability. Physical columns, exact SSE event schemas, retryable-error classification, analytics-section response contracts, observability targets, and migration details still require acceptance in the implementation-ready specs.