# Data Foundation Capability Spec - Status: Draft — not approved for implementation - Capability: Data foundation - Last updated: 2026-08-20 ## Objective Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project. ## In scope - Streamed `.csv.gz` ingestion from a local, Git-ignored input path - One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher - Import runs, checksums, schema version, row counts, rejection/error reporting - Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts - Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes - Rebuildable daily base summaries and metric-specific period projections for approved analytics - Synthetic merchant display profiles and one demo account per merchant - Idempotent re-runs and explicit import lifecycle - Database constraints/indexes driven by documented access paths - Development seed/reporting workflow and migration/rollback contract ## Out of scope - Dashboard UI - Merchant-facing API endpoints - AI recommendations - Production backup mechanics (owned by platform operations) - Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path - Treating the challenge archive or local database as version-controlled source ## Proposed logical model (technology-neutral) ```mermaid erDiagram MERCHANT_CATEGORY ||--o{ MERCHANT : classifies MERCHANT ||--o{ TERMINAL : owns MERCHANT ||--o{ MERCHANT_USER : authenticates_as MERCHANT ||--o{ PAYMENT_SESSION : receives TERMINAL ||--o{ PAYMENT_SESSION : handles PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by IMPORT_RUN ||--o{ IMPORT_ISSUE : reports MERCHANT_CATEGORY { bigint id PK text source_category_key UK text title_fa } MERCHANT { bigint id PK text source_merchant_key UK bigint category_id FK text display_name boolean profile_is_synthetic } TERMINAL { bigint id PK text source_terminal_key UK bigint merchant_id FK } MERCHANT_USER { bigint id PK bigint merchant_id FK text username UK text password_hash boolean is_demo timestamptz disabled_at } PAYMENT_SESSION { bigint id PK text source_session_key UK bigint merchant_id FK bigint terminal_id FK bigint amount text session_status timestamptz created_at bigint import_run_id FK } PAYMENT_ATTEMPT { bigint id PK bigint payment_session_id FK int try_seq text try_status text switch_response_code text psp_code text issuer_bank_code text payer_card_key text verify_type bigint adjusted_fee int init_time_ms int verify_time_ms timestamptz try_created_at timestamptz verified_at timestamptz settled_at timestamptz expires_at } IMPORT_RUN { bigint id PK text source_sha256 text source_schema_version text importer_revision text status bigint observed_rows timestamptz started_at timestamptz completed_at } IMPORT_ISSUE { bigint id PK bigint import_run_id FK text severity text rule_code bigint affected_rows jsonb evidence } CALENDAR_DATE { date date PK int jalali_year int jalali_month int jalali_day text persian_season int iso_weekday date reporting_week_start } MERCHANT_DAY_SUMMARY { bigint merchant_id FK date local_date FK bigint session_count bigint attempt_count bigint successful_count bigint amount_sum bigint source_import_run_id FK } ``` This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved. ## Import contract This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline. 1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`. 2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum. 3. Validate headers and supported schema version before database writes. 4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities. 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. 8. Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh. 9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract. ## Production write and analytics freshness contract - ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema. - This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow. - Live rows are distinguishable from bootstrap rows and include stable creation/update metadata for incremental analytics. Bootstrap-only import provenance is not forged for live rows. - Incremental analytics projections refresh within 15 minutes of an eligible source change. - A nightly idempotent reconciliation checks/recomputes the approved authoritative window and detects drift. - Failed refreshes never advance the API/UI freshness watermark. ## Synthetic merchant/account rules - Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality. - Generate stable display names from a deterministic seed so re-imports do not rename merchants. - Mark every generated profile field as synthetic in storage. - Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path. - Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks. ## Acceptance criteria - A clean database can be migrated and populated from the expected archive without loading the full file into process memory. - Re-running the same archive is safe and does not duplicate business rows. - Source row count and accepted/rejected counts reconcile exactly. - Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships. - `(source_session_key, try_seq)` is unique. - Missing values remain missing rather than being silently replaced with factual-looking defaults. - The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics. - Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked. - Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics. - Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. - Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. - Incremental projections become queryable within 15 minutes of eligible live writes, and nightly reconciliation detects/corrects drift without double counting. - Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes. ## Approval gates / unresolved decisions Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001. Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002. Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003. Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004. Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005. Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006. Resolved: use normalized source-of-truth entities plus a reporting-date dimension, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. Resolved: ZarinPal writes future production rows directly to the normalized tables. Our backend owns schema/read/analytics contracts, refreshes projections within 15 minutes, and performs nightly reconciliation. CSV ingestion remains one-time only. See ADR 0008. Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010. Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`. No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, reporting-period boundaries, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.