From 2cd53b038519f7d30ac8977e80d03ae9f6732b26 Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 12:36:32 +0330 Subject: [PATCH] docs: draft data foundation specification --- docs/DATA-FOUNDATION-SPEC.md | 156 +++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 docs/DATA-FOUNDATION-SPEC.md diff --git a/docs/DATA-FOUNDATION-SPEC.md b/docs/DATA-FOUNDATION-SPEC.md new file mode 100644 index 0000000..d11a25c --- /dev/null +++ b/docs/DATA-FOUNDATION-SPEC.md @@ -0,0 +1,156 @@ +# Data Foundation Capability Spec + +- Status: Draft — not approved for implementation +- Capability: Data foundation +- Last updated: 2026-08-20 + +## Objective + +Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. + +## In scope + +- Streamed `.csv.gz` ingestion from a local, Git-ignored input path +- Import runs, checksums, schema version, row counts, rejection/error reporting +- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts +- Synthetic merchant display profiles and one demo account per merchant +- Idempotent re-runs and explicit import lifecycle +- Database constraints/indexes driven by documented access paths +- Development seed/reporting workflow and migration/rollback contract + +## Out of scope + +- Dashboard UI +- Merchant-facing API endpoints +- AI recommendations +- Production backup mechanics (owned by platform operations) +- Treating the challenge archive or local database as version-controlled source + +## Proposed logical model (technology-neutral) + +```mermaid +erDiagram + MERCHANT_CATEGORY ||--o{ MERCHANT : classifies + MERCHANT ||--o{ TERMINAL : owns + MERCHANT ||--o{ MERCHANT_USER : authenticates_as + MERCHANT ||--o{ PAYMENT_SESSION : receives + TERMINAL ||--o{ PAYMENT_SESSION : handles + PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains + IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by + IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by + IMPORT_RUN ||--o{ IMPORT_ISSUE : reports + + MERCHANT_CATEGORY { + bigint id PK + text source_category_key UK + text title_fa + } + MERCHANT { + bigint id PK + text source_merchant_key UK + bigint category_id FK + text display_name + boolean profile_is_synthetic + } + TERMINAL { + bigint id PK + text source_terminal_key UK + bigint merchant_id FK + } + MERCHANT_USER { + bigint id PK + bigint merchant_id FK + text username UK + text password_hash + boolean is_demo + timestamptz disabled_at + } + PAYMENT_SESSION { + bigint id PK + text source_session_key UK + bigint merchant_id FK + bigint terminal_id FK + bigint amount + text session_status + timestamptz created_at + bigint import_run_id FK + } + PAYMENT_ATTEMPT { + bigint id PK + bigint payment_session_id FK + int try_seq + text try_status + text switch_response_code + text psp_code + text issuer_bank_code + text payer_card_key + text verify_type + bigint adjusted_fee + int init_time_ms + int verify_time_ms + timestamptz try_created_at + timestamptz verified_at + timestamptz settled_at + int expire_in + } + IMPORT_RUN { + bigint id PK + text source_sha256 + text source_schema_version + text importer_revision + text status + bigint observed_rows + timestamptz started_at + timestamptz completed_at + } + IMPORT_ISSUE { + bigint id PK + bigint import_run_id FK + text severity + text rule_code + bigint affected_rows + jsonb evidence + } +``` + +This ERD is a logical baseline, not an approved physical schema. In particular, source key types, timestamp timezone interpretation, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved. + +## Import contract + +1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`. +2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum. +3. Validate headers and supported schema version before database writes. +4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities. +5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. +6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. +7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. + +## Synthetic merchant/account rules + +- Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality. +- Generate stable display names from a deterministic seed so re-imports do not rename merchants. +- Mark every generated profile field as synthetic in storage. +- Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path. +- Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks. + +## Acceptance criteria + +- A clean database can be migrated and populated from the expected archive without loading the full file into process memory. +- Re-running the same archive is safe and does not duplicate business rows. +- Source row count and accepted/rejected counts reconcile exactly. +- Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships. +- `(source_session_key, try_seq)` is unique. +- Missing values remain missing rather than being silently replaced with factual-looking defaults. +- The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics. +- Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked. + +## Approval gates / unresolved decisions + +1. Go architecture style: vertical-slice/modular monolith, hexagonal, clean architecture, or another preference. +2. Dependency injection: manual constructor injection, compile-time/runtime DI library, or none. +3. PostgreSQL access stack: ORM choice plus a bulk-ingest path suitable for 2.2M+ rows. +4. Source timestamp timezone and daylight-saving assumptions. +5. Whether `payer_card_key` must be retained, irreversibly transformed, or omitted after derived analytics. +6. Minimum cohort size and suppression policy for peer benchmarks. +7. Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone. +