Data Foundation Capability Spec

Status: Draft — not approved for implementation
Capability: Data foundation
Last updated: 2026-08-20

Objective

Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project.

In scope

Streamed .csv.gz ingestion from a local, Git-ignored input path
One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher
Import runs, checksums, schema version, row counts, rejection/error reporting
Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
Rebuildable daily base summaries and metric-specific period projections for approved analytics
Synthetic merchant display profiles and one demo account per merchant
Idempotent re-runs and explicit import lifecycle
Database constraints/indexes driven by documented access paths
Development seed/reporting workflow and migration/rollback contract

Out of scope

Dashboard UI
Merchant-facing API endpoints
AI recommendations
Production backup mechanics (owned by platform operations)
Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path
Treating the challenge archive or local database as version-controlled source

Proposed logical model (technology-neutral)

erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups
    MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      bigint adjusted_fee
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      timestamptz expires_at
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }
    CALENDAR_DATE {
      date date PK
      int jalali_year
      int jalali_month
      int jalali_day
      text persian_season
      int iso_weekday
      date reporting_week_start
      date reporting_week_end
    }
    MERCHANT_DAY_SUMMARY {
      bigint merchant_id FK
      date local_date FK
      bigint session_count
      bigint attempt_count
      bigint successful_count
      bigint amount_sum
      bigint source_import_run_id FK
    }

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

Import contract

This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline.

Accept an explicit path, defaulting locally to data/incoming/challenge_data.csv.gz.
Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
Validate headers and supported schema version before database writes.
Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
Validate uniqueness of (session_key, try_seq), referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard projections; first relevant access owns lazy calculation.
Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions invalidate affected lazy analytics projections.

Production write and lazy analytics-cache contract

ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
The first relevant dashboard request calculates typed merchant analytics projections and persists them in PostgreSQL. Later requests reuse projections for the same metric-definition and source revisions.
Every inserted related payment fact atomically advances the affected merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
A request with missing or stale projections obtains a merchant-scoped PostgreSQL lock, recalculates once, publishes atomically, and records the exact source/cohort and metric-definition revisions represented. Concurrent requests reuse the completed result rather than duplicate expensive queries.
Invalidation marks cached results stale through revision mismatch; it does not synchronously recalculate or delete the last successful snapshot.
Failed calculations never mark a revision current. The API may expose the prior snapshot only with an explicit stale warning and last-calculated metadata.
No scheduled analytics refresh worker, nightly reconciliation job, Redis, or time-based expiry is required for the static MVP dataset.

Synthetic merchant/account rules

Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
Generate stable display names from a deterministic seed so re-imports do not rename merchants.
Mark every generated profile field as synthetic in storage.
Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

Acceptance criteria

A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
Re-running the same archive is safe and does not duplicate business rows.
Source row count and accepted/rejected counts reconcile exactly.
Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
(source_session_key, try_seq) is unique.
Missing values remain missing rather than being silently replaced with factual-looking defaults.
The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics.
Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
First access creates reusable merchant-scoped projections; a second equivalent request reads the current cache without repeating source-fact aggregation.
Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
Concurrent stale-cache requests do not duplicate the same merchant calculation, and failed calculation does not claim the new revision is cached.
Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes.
Every reporting week starts Saturday at 00:00:00 and ends Friday at 23:59:59.999… in Asia/Tehran; API ranges use an equivalent half-open [start, next_start) interval.

Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin cmd/ composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL COPY for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003.

Resolved: interpret timezone-naive source timestamps as Asia/Tehran, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.

Resolved: retain payer_card_key unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.

Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006.

Resolved: use normalized source-of-truth entities plus a reporting-date dimension, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.

Resolved: ZarinPal writes future append-only production rows directly to normalized tables. Analytics are calculated lazily on first relevant dashboard access, stored as typed PostgreSQL projections, and invalidated through merchant/cohort source revisions advanced on insert. No scheduled refresh or Redis is required. See ADR 0011.

Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.

Resolved: reporting weeks follow the Iranian week: Saturday through Friday in Asia/Tehran, represented as half-open API/database ranges. See ADR 0012.

Resolved: only source session status Verified is a definitely successful payment. Keep Paid separate until an authoritative ZarinPal definition changes the metric catalogue. See METRIC-CATALOGUE.md.

No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, reporting-period boundaries, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.

13 KiB Raw Blame History