Data Foundation Capability Spec

Status: Draft — not approved for implementation
Capability: Data foundation
Last updated: 2026-08-20

Objective

Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project.

In scope

Streamed .csv.gz ingestion from a local, Git-ignored input path
One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher
Import runs, checksums, schema version, row counts, rejection/error reporting
Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
Versioned Gregorian/Jalali calendar dates and curated Iranian event/holiday periods
Rebuildable daily base summaries and metric-specific period projections for approved analytics
Synthetic merchant display profiles and one demo account per merchant
Idempotent re-runs and explicit import lifecycle
Database constraints/indexes driven by documented access paths
Development seed/reporting workflow and migration/rollback contract

Out of scope

Dashboard UI
Merchant-facing API endpoints
AI recommendations
Production backup mechanics (owned by platform operations)
Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path
Treating the challenge archive or local database as version-controlled source

Proposed logical model (technology-neutral)

erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    CALENDAR_DATE ||--o{ CALENDAR_DATE_EVENT : includes
    CALENDAR_EVENT ||--o{ CALENDAR_DATE_EVENT : occurs_on
    CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups
    MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      bigint adjusted_fee
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      timestamptz expires_at
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }
    CALENDAR_DATE {
      date date PK
      int jalali_year
      int jalali_month
      int jalali_day
      text persian_season
      int iso_weekday
    }
    CALENDAR_EVENT {
      bigint id PK
      text code UK
      text title_fa
      text event_type
      text source_version
    }
    CALENDAR_DATE_EVENT {
      date date FK
      bigint calendar_event_id FK
    }
    MERCHANT_DAY_SUMMARY {
      bigint merchant_id FK
      date local_date FK
      bigint session_count
      bigint attempt_count
      bigint successful_count
      bigint amount_sum
      bigint source_import_run_id FK
    }

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

Import contract

This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline.

Accept an explicit path, defaulting locally to data/incoming/challenge_data.csv.gz.
Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
Validate headers and supported schema version before database writes.
Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
Validate uniqueness of (session_key, try_seq), referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.
Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract.

Production write and analytics freshness contract

ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
Live rows are distinguishable from bootstrap rows and include stable creation/update metadata for incremental analytics. Bootstrap-only import provenance is not forged for live rows.
Incremental analytics projections refresh within 15 minutes of an eligible source change.
A nightly idempotent reconciliation checks/recomputes the approved authoritative window and detects drift.
Failed refreshes never advance the API/UI freshness watermark.

Synthetic merchant/account rules

Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
Generate stable display names from a deterministic seed so re-imports do not rename merchants.
Mark every generated profile field as synthetic in storage.
Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

Acceptance criteria

A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
Re-running the same archive is safe and does not duplicate business rows.
Source row count and accepted/rejected counts reconcile exactly.
Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
(source_session_key, try_seq) is unique.
Missing values remain missing rather than being silently replaced with factual-looking defaults.
The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
Daily, Jalali-month, Persian-season, and approved calendar-event analytics reconcile to deterministic source queries for additive metrics.
Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
Incremental projections become queryable within 15 minutes of eligible live writes, and nightly reconciliation detects/corrects drift without double counting.
Calendar rows covering the reporting period reconcile with reviewed official snapshots and record provenance/checksums.

Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin cmd/ composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL COPY for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003.

Resolved: interpret timezone-naive source timestamps as Asia/Tehran, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.

Resolved: retain payer_card_key unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.

Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006.

Resolved: use normalized source-of-truth entities plus versioned calendar/event dimensions, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Monthly, Persian-season, and holiday results use explicit calendar definitions. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.

Resolved: ZarinPal writes future production rows directly to the normalized tables. Our backend owns schema/read/analytics contracts, refreshes projections within 15 minutes, and performs nightly reconciliation. CSV ingestion remains one-time only. See ADR 0008.

Resolved: commit reviewed yearly Iranian calendar snapshots sourced from the University of Tehran Institute of Geophysics Calendar Center; do not call a calendar API at runtime. Official holidays and analytical campaign/event windows remain distinguishable and versioned. See ADR 0009.

Resolved: only source session status Verified is a definitely successful payment. Keep Paid separate until an authoritative ZarinPal definition changes the metric catalogue. See METRIC-CATALOGUE.md.

No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, calendar source, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.

12 KiB Raw Blame History