Data Foundation Capability Spec

Status: Draft — not approved for implementation
Capability: Data foundation
Last updated: 2026-08-20

Objective

Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics.

In scope

Streamed .csv.gz ingestion from a local, Git-ignored input path
Import runs, checksums, schema version, row counts, rejection/error reporting
Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
Synthetic merchant display profiles and one demo account per merchant
Idempotent re-runs and explicit import lifecycle
Database constraints/indexes driven by documented access paths
Development seed/reporting workflow and migration/rollback contract

Out of scope

Dashboard UI
Merchant-facing API endpoints
AI recommendations
Production backup mechanics (owned by platform operations)
Treating the challenge archive or local database as version-controlled source

Proposed logical model (technology-neutral)

erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      bigint adjusted_fee
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      int expire_in
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, timestamp timezone interpretation, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

Import contract

Accept an explicit path, defaulting locally to data/incoming/challenge_data.csv.gz.
Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
Validate headers and supported schema version before database writes.
Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
Validate uniqueness of (session_key, try_seq), referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.

Synthetic merchant/account rules

Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
Generate stable display names from a deterministic seed so re-imports do not rename merchants.
Mark every generated profile field as synthetic in storage.
Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

Acceptance criteria

A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
Re-running the same archive is safe and does not duplicate business rows.
Source row count and accepted/rejected counts reconcile exactly.
Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
(source_session_key, try_seq) is unique.
Missing values remain missing rather than being silently replaced with factual-looking defaults.
The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.

Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin cmd/ composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL COPY for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003.

Remaining approval gates:

Source timestamp timezone and daylight-saving assumptions.
Whether payer_card_key must be retained, irreversibly transformed, or omitted after derived analytics.
Minimum cohort size and suppression policy for peer benchmarks.
Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone.

6.6 KiB Raw Blame History