zarinpal-analytics-backend/docs/DATA-FOUNDATION-SPEC.md

# Data Foundation Capability Spec

- Status: Draft — not approved for implementation
- Capability: Data foundation
- Last updated: 2026-08-20

## Objective

Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics.

## In scope

- Streamed `.csv.gz` ingestion from a local, Git-ignored input path
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
- Development seed/reporting workflow and migration/rollback contract

## Out of scope

- Dashboard UI
- Merchant-facing API endpoints
- AI recommendations
- Production backup mechanics (owned by platform operations)
- Treating the challenge archive or local database as version-controlled source

## Proposed logical model (technology-neutral)

```mermaid
erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      bigint adjusted_fee
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      int expire_in
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }
```

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, timestamp timezone interpretation, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

## Import contract

1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`.
2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
3. Validate headers and supported schema version before database writes.
4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.

## Synthetic merchant/account rules

- Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
- Generate stable display names from a deterministic seed so re-imports do not rename merchants.
- Mark every generated profile field as synthetic in storage.
- Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
- Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

## Acceptance criteria

- A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
- Re-running the same archive is safe and does not duplicate business rows.
- Source row count and accepted/rejected counts reconcile exactly.
- Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
- `(source_session_key, try_seq)` is unique.
- Missing values remain missing rather than being silently replaced with factual-looking defaults.
- The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
- Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.

## Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003.

Remaining approval gates:

1. Source timestamp timezone and daylight-saving assumptions.
2. Whether `payer_card_key` must be retained, irreversibly transformed, or omitted after derived analytics.
3. Minimum cohort size and suppression policy for peer benchmarks.
4. Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone.