# Data Foundation Capability Spec - Status: Draft — not approved for implementation - Capability: Data foundation - Last updated: 2026-08-20 ## Objective Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. ## In scope - Streamed `.csv.gz` ingestion from a local, Git-ignored input path - Import runs, checksums, schema version, row counts, rejection/error reporting - Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts - Versioned Gregorian/Jalali calendar dates and curated Iranian event/holiday periods - Rebuildable daily base summaries and metric-specific period projections for approved analytics - Synthetic merchant display profiles and one demo account per merchant - Idempotent re-runs and explicit import lifecycle - Database constraints/indexes driven by documented access paths - Development seed/reporting workflow and migration/rollback contract ## Out of scope - Dashboard UI - Merchant-facing API endpoints - AI recommendations - Production backup mechanics (owned by platform operations) - Treating the challenge archive or local database as version-controlled source ## Proposed logical model (technology-neutral) ```mermaid erDiagram MERCHANT_CATEGORY ||--o{ MERCHANT : classifies MERCHANT ||--o{ TERMINAL : owns MERCHANT ||--o{ MERCHANT_USER : authenticates_as MERCHANT ||--o{ PAYMENT_SESSION : receives TERMINAL ||--o{ PAYMENT_SESSION : handles PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains CALENDAR_DATE ||--o{ CALENDAR_DATE_EVENT : includes CALENDAR_EVENT ||--o{ CALENDAR_DATE_EVENT : occurs_on CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by IMPORT_RUN ||--o{ IMPORT_ISSUE : reports MERCHANT_CATEGORY { bigint id PK text source_category_key UK text title_fa } MERCHANT { bigint id PK text source_merchant_key UK bigint category_id FK text display_name boolean profile_is_synthetic } TERMINAL { bigint id PK text source_terminal_key UK bigint merchant_id FK } MERCHANT_USER { bigint id PK bigint merchant_id FK text username UK text password_hash boolean is_demo timestamptz disabled_at } PAYMENT_SESSION { bigint id PK text source_session_key UK bigint merchant_id FK bigint terminal_id FK bigint amount text session_status timestamptz created_at bigint import_run_id FK } PAYMENT_ATTEMPT { bigint id PK bigint payment_session_id FK int try_seq text try_status text switch_response_code text psp_code text issuer_bank_code text payer_card_key text verify_type bigint adjusted_fee int init_time_ms int verify_time_ms timestamptz try_created_at timestamptz verified_at timestamptz settled_at int expire_in } IMPORT_RUN { bigint id PK text source_sha256 text source_schema_version text importer_revision text status bigint observed_rows timestamptz started_at timestamptz completed_at } IMPORT_ISSUE { bigint id PK bigint import_run_id FK text severity text rule_code bigint affected_rows jsonb evidence } CALENDAR_DATE { date date PK int jalali_year int jalali_month int jalali_day text persian_season int iso_weekday } CALENDAR_EVENT { bigint id PK text code UK text title_fa text event_type text source_version } CALENDAR_DATE_EVENT { date date FK bigint calendar_event_id FK } MERCHANT_DAY_SUMMARY { bigint merchant_id FK date local_date FK bigint session_count bigint attempt_count bigint successful_count bigint amount_sum bigint source_import_run_id FK } ``` This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved. ## Import contract 1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`. 2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum. 3. Validate headers and supported schema version before database writes. 4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities. 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. 8. Populate and validate the approved calendar range, then rebuild affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh. ## Synthetic merchant/account rules - Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality. - Generate stable display names from a deterministic seed so re-imports do not rename merchants. - Mark every generated profile field as synthetic in storage. - Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path. - Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks. ## Acceptance criteria - A clean database can be migrated and populated from the expected archive without loading the full file into process memory. - Re-running the same archive is safe and does not duplicate business rows. - Source row count and accepted/rejected counts reconcile exactly. - Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships. - `(source_session_key, try_seq)` is unique. - Missing values remain missing rather than being silently replaced with factual-looking defaults. - The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics. - Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked. - Daily, Jalali-month, Persian-season, and approved calendar-event analytics reconcile to deterministic source queries for additive metrics. - Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. - Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. ## Approval gates / unresolved decisions Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001. Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002. Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003. Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004. Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005. Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006. Resolved: use normalized source-of-truth entities plus versioned calendar/event dimensions, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Monthly, Persian-season, and holiday results use explicit calendar definitions. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, calendar source, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.