# Data Foundation Capability Spec - Status: Draft — not approved for implementation - Capability: Data foundation - Last updated: 2026-08-20 ## Objective Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project. ## In scope - Streamed `.csv.gz` ingestion from a local, Git-ignored input path - One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher - Import runs, checksums, schema version, row counts, rejection/error reporting - Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts - Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes - PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys - Synthetic merchant display profiles and one demo account per merchant - Idempotent re-runs and explicit import lifecycle - Database constraints/indexes driven by documented access paths - Development seed/reporting workflow and migration/rollback contract ## Out of scope - Dashboard UI - Merchant-facing API endpoints - AI recommendations - Production backup mechanics (owned by platform operations) - Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path - Treating the challenge archive or local database as version-controlled source ## Proposed logical model (technology-neutral) ```mermaid erDiagram MERCHANT_CATEGORY ||--o{ MERCHANT : classifies MERCHANT ||--o{ TERMINAL : owns MERCHANT ||--o{ MERCHANT_USER : authenticates_as MERCHANT ||--o{ PAYMENT_SESSION : receives TERMINAL ||--o{ PAYMENT_SESSION : handles PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by IMPORT_RUN ||--o{ IMPORT_ISSUE : reports MERCHANT_CATEGORY { bigint id PK text source_category_key UK text title_fa } MERCHANT { bigint id PK text source_merchant_key UK bigint category_id FK text display_name boolean profile_is_synthetic } TERMINAL { bigint id PK text source_terminal_key UK bigint merchant_id FK } MERCHANT_USER { bigint id PK bigint merchant_id FK text username UK text password_hash boolean is_demo timestamptz disabled_at } PAYMENT_SESSION { bigint id PK text source_session_key UK bigint merchant_id FK bigint terminal_id FK bigint amount text session_status timestamptz created_at bigint import_run_id FK } PAYMENT_ATTEMPT { bigint id PK bigint payment_session_id FK int try_seq text try_status text switch_response_code text psp_code text issuer_bank_code text payer_card_key text verify_type bigint adjusted_fee int init_time_ms int verify_time_ms timestamptz try_created_at timestamptz verified_at timestamptz settled_at timestamptz expires_at } IMPORT_RUN { bigint id PK text source_sha256 text source_schema_version text importer_revision text status bigint observed_rows timestamptz started_at timestamptz completed_at } IMPORT_ISSUE { bigint id PK bigint import_run_id FK text severity text rule_code bigint affected_rows jsonb evidence } CALENDAR_DATE { date date PK int jalali_year int jalali_month int jalali_day text persian_season int iso_weekday date reporting_week_start date reporting_week_end } ``` This ERD is a logical baseline, not an approved physical schema. In particular, source key types, cache/invalidation state, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved. ## Import contract This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline. 1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`. 2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum. 3. Validate headers and supported schema version before database writes. 4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities. 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. 8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs. 9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete. ## Production write and lazy analytics-cache contract - ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema. - This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow. - Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract. - An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. - Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL. - A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish. - Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant. - Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers. - Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan. - Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available. - Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation. - Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details. ## Synthetic merchant/account rules - Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality. - Generate stable display names from a deterministic seed so re-imports do not rename merchants. - Mark every generated profile field as synthetic in storage. - Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path. - Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks. ## Acceptance criteria - A clean database can be migrated and populated from the expected archive without loading the full file into process memory. - Re-running the same archive is safe and does not duplicate business rows. - Source row count and accepted/rejected counts reconcile exactly. - Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships. - `(source_session_key, try_seq)` is unique. - Missing values remain missing rather than being silently replaced with factual-looking defaults. - The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics. - Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked. - Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics. - Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. - Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. - First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready. - A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation. - Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically. - Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out. - One failed section does not hide ready sections and does not create a ready cache entry. - A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately. - Redis loss produces a cold-cache calculation rather than loss of source data. - Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes. - Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval. ## Approval gates / unresolved decisions Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001. Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002. Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003. Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004. Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005. Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006. Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four parallel sections in a separate Go worker; Redis caches each typed result for 30 days; an authenticated `GET` SSE stream reveals sections as ready. Each section has a 60-second attempt timeout and at most two transient retries using exponential backoff/jitter. PostgreSQL merchant/cohort revisions provide logical invalidation. See ADR 0011. Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010. Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `Asia/Tehran`, represented as half-open API/database ranges. See ADR 0012. Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`. No broad architecture approval gates remain for this capability. Physical columns, indexes, exact SSE schemas/reconnect retention, worker concurrency, retry classification, analytics-section response contracts, and migration details still require acceptance in the implementation-ready specs.