zarinpal-analytics-backend/docs/DATA-FOUNDATION-SPEC.md

# Data Foundation Capability Spec

- Status: Draft — not approved for implementation
- Capability: Data foundation
- Last updated: 2026-08-20

## Objective

Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics.

## In scope

- Streamed `.csv.gz` ingestion from a local, Git-ignored input path
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Versioned Gregorian/Jalali calendar dates and curated Iranian event/holiday periods
- Rebuildable daily base summaries and metric-specific period projections for approved analytics
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
- Development seed/reporting workflow and migration/rollback contract

## Out of scope

- Dashboard UI
- Merchant-facing API endpoints
- AI recommendations
- Production backup mechanics (owned by platform operations)
- Treating the challenge archive or local database as version-controlled source

## Proposed logical model (technology-neutral)

```mermaid
erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    CALENDAR_DATE ||--o{ CALENDAR_DATE_EVENT : includes
    CALENDAR_EVENT ||--o{ CALENDAR_DATE_EVENT : occurs_on
    CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups
    MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      bigint adjusted_fee
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      int expire_in
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }
    CALENDAR_DATE {
      date date PK
      int jalali_year
      int jalali_month
      int jalali_day
      text persian_season
      int iso_weekday
    }
    CALENDAR_EVENT {
      bigint id PK
      text code UK
      text title_fa
      text event_type
      text source_version
    }
    CALENDAR_DATE_EVENT {
      date date FK
      bigint calendar_event_id FK
    }
    MERCHANT_DAY_SUMMARY {
      bigint merchant_id FK
      date local_date FK
      bigint session_count
      bigint attempt_count
      bigint successful_count
      bigint amount_sum
      bigint source_import_run_id FK
    }
```

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

## Import contract

1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`.
2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
3. Validate headers and supported schema version before database writes.
4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved calendar range, then rebuild affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.

## Synthetic merchant/account rules

- Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
- Generate stable display names from a deterministic seed so re-imports do not rename merchants.
- Mark every generated profile field as synthetic in storage.
- Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
- Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

## Acceptance criteria

- A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
- Re-running the same archive is safe and does not duplicate business rows.
- Source row count and accepted/rejected counts reconcile exactly.
- Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
- `(source_session_key, try_seq)` is unique.
- Missing values remain missing rather than being silently replaced with factual-looking defaults.
- The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
- Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
- Daily, Jalali-month, Persian-season, and approved calendar-event analytics reconcile to deterministic source queries for additive metrics.
- Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
- Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.

## Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003.

Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.

Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.

Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006.

Resolved: use normalized source-of-truth entities plus versioned calendar/event dimensions, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Monthly, Persian-season, and holiday results use explicit calendar definitions. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.

No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, calendar source, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.