docs: draft data foundation specification
This commit is contained in:
156
docs/DATA-FOUNDATION-SPEC.md
Normal file
156
docs/DATA-FOUNDATION-SPEC.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Data Foundation Capability Spec
|
||||
|
||||
- Status: Draft — not approved for implementation
|
||||
- Capability: Data foundation
|
||||
- Last updated: 2026-08-20
|
||||
|
||||
## Objective
|
||||
|
||||
Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics.
|
||||
|
||||
## In scope
|
||||
|
||||
- Streamed `.csv.gz` ingestion from a local, Git-ignored input path
|
||||
- Import runs, checksums, schema version, row counts, rejection/error reporting
|
||||
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
|
||||
- Synthetic merchant display profiles and one demo account per merchant
|
||||
- Idempotent re-runs and explicit import lifecycle
|
||||
- Database constraints/indexes driven by documented access paths
|
||||
- Development seed/reporting workflow and migration/rollback contract
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Dashboard UI
|
||||
- Merchant-facing API endpoints
|
||||
- AI recommendations
|
||||
- Production backup mechanics (owned by platform operations)
|
||||
- Treating the challenge archive or local database as version-controlled source
|
||||
|
||||
## Proposed logical model (technology-neutral)
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
MERCHANT_CATEGORY ||--o{ MERCHANT : classifies
|
||||
MERCHANT ||--o{ TERMINAL : owns
|
||||
MERCHANT ||--o{ MERCHANT_USER : authenticates_as
|
||||
MERCHANT ||--o{ PAYMENT_SESSION : receives
|
||||
TERMINAL ||--o{ PAYMENT_SESSION : handles
|
||||
PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
|
||||
IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
|
||||
IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
|
||||
IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
|
||||
|
||||
MERCHANT_CATEGORY {
|
||||
bigint id PK
|
||||
text source_category_key UK
|
||||
text title_fa
|
||||
}
|
||||
MERCHANT {
|
||||
bigint id PK
|
||||
text source_merchant_key UK
|
||||
bigint category_id FK
|
||||
text display_name
|
||||
boolean profile_is_synthetic
|
||||
}
|
||||
TERMINAL {
|
||||
bigint id PK
|
||||
text source_terminal_key UK
|
||||
bigint merchant_id FK
|
||||
}
|
||||
MERCHANT_USER {
|
||||
bigint id PK
|
||||
bigint merchant_id FK
|
||||
text username UK
|
||||
text password_hash
|
||||
boolean is_demo
|
||||
timestamptz disabled_at
|
||||
}
|
||||
PAYMENT_SESSION {
|
||||
bigint id PK
|
||||
text source_session_key UK
|
||||
bigint merchant_id FK
|
||||
bigint terminal_id FK
|
||||
bigint amount
|
||||
text session_status
|
||||
timestamptz created_at
|
||||
bigint import_run_id FK
|
||||
}
|
||||
PAYMENT_ATTEMPT {
|
||||
bigint id PK
|
||||
bigint payment_session_id FK
|
||||
int try_seq
|
||||
text try_status
|
||||
text switch_response_code
|
||||
text psp_code
|
||||
text issuer_bank_code
|
||||
text payer_card_key
|
||||
text verify_type
|
||||
bigint adjusted_fee
|
||||
int init_time_ms
|
||||
int verify_time_ms
|
||||
timestamptz try_created_at
|
||||
timestamptz verified_at
|
||||
timestamptz settled_at
|
||||
int expire_in
|
||||
}
|
||||
IMPORT_RUN {
|
||||
bigint id PK
|
||||
text source_sha256
|
||||
text source_schema_version
|
||||
text importer_revision
|
||||
text status
|
||||
bigint observed_rows
|
||||
timestamptz started_at
|
||||
timestamptz completed_at
|
||||
}
|
||||
IMPORT_ISSUE {
|
||||
bigint id PK
|
||||
bigint import_run_id FK
|
||||
text severity
|
||||
text rule_code
|
||||
bigint affected_rows
|
||||
jsonb evidence
|
||||
}
|
||||
```
|
||||
|
||||
This ERD is a logical baseline, not an approved physical schema. In particular, source key types, timestamp timezone interpretation, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.
|
||||
|
||||
## Import contract
|
||||
|
||||
1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`.
|
||||
2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
|
||||
3. Validate headers and supported schema version before database writes.
|
||||
4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
|
||||
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
|
||||
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
|
||||
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
|
||||
|
||||
## Synthetic merchant/account rules
|
||||
|
||||
- Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality.
|
||||
- Generate stable display names from a deterministic seed so re-imports do not rename merchants.
|
||||
- Mark every generated profile field as synthetic in storage.
|
||||
- Never derive passwords from merchant keys. Generate or seed credentials through an explicit one-off command and display/export them only through an approved secure path.
|
||||
- Password storage and session behavior belong to the identity spec; this capability only establishes referential entities and safe seeding hooks.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A clean database can be migrated and populated from the expected archive without loading the full file into process memory.
|
||||
- Re-running the same archive is safe and does not duplicate business rows.
|
||||
- Source row count and accepted/rejected counts reconcile exactly.
|
||||
- Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
|
||||
- `(source_session_key, try_seq)` is unique.
|
||||
- Missing values remain missing rather than being silently replaced with factual-looking defaults.
|
||||
- The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
|
||||
- Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
|
||||
|
||||
## Approval gates / unresolved decisions
|
||||
|
||||
1. Go architecture style: vertical-slice/modular monolith, hexagonal, clean architecture, or another preference.
|
||||
2. Dependency injection: manual constructor injection, compile-time/runtime DI library, or none.
|
||||
3. PostgreSQL access stack: ORM choice plus a bulk-ingest path suitable for 2.2M+ rows.
|
||||
4. Source timestamp timezone and daylight-saving assumptions.
|
||||
5. Whether `payer_card_key` must be retained, irreversibly transformed, or omitted after derived analytics.
|
||||
6. Minimum cohort size and suppression policy for peer benchmarks.
|
||||
7. Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone.
|
||||
|
||||
Reference in New Issue
Block a user