Data Foundation Capability Spec

Status: Draft — not approved for implementation
Capability: Data foundation
Last updated: 2026-08-20

Objective

Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project.

In scope

Streamed .csv.gz ingestion from a local, Git-ignored input path
One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher
Import runs, checksums, schema version, row counts, rejection/error reporting
Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
Synthetic merchant display profiles and one demo account per merchant
Idempotent re-runs and explicit import lifecycle
Database constraints/indexes driven by documented access paths
Development seed/reporting workflow and migration/rollback contract

Out of scope

Dashboard UI
Merchant-facing API endpoints
AI recommendations
Production backup mechanics (owned by platform operations)
Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path
Treating the challenge archive or local database as version-controlled source

Proposed logical model (technology-neutral)

erDiagram
    MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
    MERCHANT ||--o{ TERMINAL : owns
    MERCHANT ||--o{ MERCHANT_USER : authenticates_as
    MERCHANT ||--o{ PAYMENT_SESSION : receives
    TERMINAL ||--o{ PAYMENT_SESSION : handles
    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
    IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
    IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
    IMPORT_RUN ||--o{ IMPORT_ISSUE : reports

    MERCHANT_CATEGORY {
      bigint id PK
      text source_category_key UK
      text title_fa
    }
    MERCHANT {
      bigint id PK
      text source_merchant_key UK
      bigint category_id FK
      text display_name
      boolean profile_is_synthetic
    }
    TERMINAL {
      bigint id PK
      text source_terminal_key UK
      bigint merchant_id FK
    }
    MERCHANT_USER {
      bigint id PK
      bigint merchant_id FK
      text username UK
      text password_hash
      boolean is_demo
      timestamptz disabled_at
    }
    PAYMENT_SESSION {
      bigint id PK
      text source_session_key UK
      bigint merchant_id FK
      bigint terminal_id FK
      bigint amount
      bigint fee_amount_rial
      text session_status
      timestamptz created_at
      bigint import_run_id FK
    }
    PAYMENT_ATTEMPT {
      bigint id PK
      bigint payment_session_id FK
      int try_seq
      text try_status
      text switch_response_code
      text psp_code
      text issuer_bank_code
      text payer_card_key
      text verify_type
      int init_time_ms
      int verify_time_ms
      timestamptz try_created_at
      timestamptz verified_at
      timestamptz settled_at
      timestamptz expires_at
    }
    IMPORT_RUN {
      bigint id PK
      text source_sha256
      text source_schema_version
      text importer_revision
      text status
      bigint observed_rows
      timestamptz started_at
      timestamptz completed_at
    }
    IMPORT_ISSUE {
      bigint id PK
      bigint import_run_id FK
      text severity
      text rule_code
      bigint affected_rows
      jsonb evidence
    }
    CALENDAR_DATE {
      date date PK
      int jalali_year
      int jalali_month
      int jalali_day
      text persian_season
      int iso_weekday
      date reporting_week_start
      date reporting_week_end
    }

This ERD is a logical baseline, not an approved physical schema. In particular, source key types, cache/invalidation state, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.

Import contract

This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline.

Accept an explicit path, defaulting locally to data/incoming/challenge_data.csv.gz.
Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
Validate headers and supported schema version before database writes.
Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
Validate uniqueness of (session_key, try_seq), referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.

Production write and lazy analytics-cache contract

ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
An idempotent authenticated POST starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—overview, payment_performance, fees, and peer_comparison—for bounded parallel calculation by a separate Go worker service.
Use Asynq pinned to v0.26.0 as the Redis-backed task queue. Asynq is pre-v1.0.0; never use a floating version, and review API/behavior compatibility before upgrading the pin.
Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL.
A separate authenticated GET SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings.
Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available.
Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return 429 with Retry-After and do not enqueue work.
SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream.
Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections.
Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.

Synthetic merchant/account rules

Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
Generate stable display names from a deterministic seed so re-imports do not rename merchants.
Mark every generated profile field as synthetic in storage.
Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.

Acceptance criteria

A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
Re-running the same archive is safe and does not duplicate business rows.
Source row count and accepted/rejected counts reconcile exactly.
Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
(source_session_key, try_seq) is unique.
Missing values remain missing rather than being silently replaced with factual-looking defaults.
The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics.
Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
Source adjusted_fee is normalized once per session as fee_amount_rial; retry attempts never multiply fee totals.
First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready.
A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation.
Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
One failed section does not hide ready sections and does not create a ready cache entry.
A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately.
Start, manual-retry, and SSE connection limits return a typed 429 with Retry-After; rejected requests enqueue no jobs and cannot affect another merchant.
SSE emits heartbeats within 15 seconds and supports authorized replay for one hour; replay expiry falls back to the current run snapshot without duplicating work.
Redis loss produces a cold-cache calculation rather than loss of source data.
Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes.
Every reporting week starts Saturday at 00:00:00 and ends Friday at 23:59:59.999… in Asia/Tehran; API ranges use an equivalent half-open [start, next_start) interval.

Approval gates / unresolved decisions

Resolved: use a capability-oriented modular monolith, with lightweight ports/adapters at external boundaries such as PostgreSQL and AvalAI. See ADR 0001.

Resolved: use manual constructor injection with explicit wiring at thin cmd/ composition roots. See ADR 0002.

Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL COPY for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003.

Resolved: interpret timezone-naive source timestamps as Asia/Tehran, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.

Resolved: retain payer_card_key unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.

Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006.

Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.

Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent POST starts four sections through Asynq v0.26.0 (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011.

Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.

Resolved: reporting weeks follow the Iranian week: Saturday through Friday in Asia/Tehran, represented as half-open API/database ranges. See ADR 0012.

Resolved: authoritative ZarinPal lifecycle guidance defines both Paid and Verified as successful bank payments. Paid means funds were deducted but merchant verification is incomplete; Verified is finalized. Success/paid-volume/recovery metrics include both while verification metrics keep them distinct. Source adjusted_fee is treated as the real merchant fee, normalized once per session, and shown in Rial. See ADR 0013 and METRIC-CATALOGUE.md.

No broad architecture approval gates remain for this capability. Physical columns, exact SSE event schemas, retryable-error classification, analytics-section response contracts, observability targets, and migration details still require acceptance in the implementation-ready specs.

16 KiB Raw Blame History