starcoach-hackathon/zarinpal-analytics-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: define bootstrap and schema proposal

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 13:47:22 +03:30
parent b14807e480
commit 5e83da4a19
4 changed files with 94 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
docs/DATA-FOUNDATION-SPEC.md
View File

@@ -6,11 +6,12 @@
## Objective
Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics.
Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project.
## In scope
- Streamed `.csv.gz` ingestion from a local, Git-ignored input path
- One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Versioned Gregorian/Jalali calendar dates and curated Iranian event/holiday periods
@@ -26,6 +27,7 @@ Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve
- Merchant-facing API endpoints
- AI recommendations
- Production backup mechanics (owned by platform operations)
- Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path
- Treating the challenge archive or local database as version-controlled source
## Proposed logical model (technology-neutral)
@@ -97,7 +99,7 @@ erDiagram
timestamptz try_created_at
timestamptz verified_at
timestamptz settled_at
int expire_in
timestamptz expires_at
}
IMPORT_RUN {
bigint id PK
@@ -151,6 +153,8 @@ This ERD is a logical baseline, not an approved physical schema. In particular,
## Import contract
This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline.
1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`.
2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
3. Validate headers and supported schema version before database writes.
@@ -158,7 +162,8 @@ This ERD is a logical baseline, not an approved physical schema. In particular,
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved calendar range, then rebuild affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.
8. Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract.
## Synthetic merchant/account rules

84
docs/PHYSICAL-SCHEMA-PROPOSAL.md Normal file
View File

@@ -0,0 +1,84 @@
# PostgreSQL Physical Schema Proposal
- Status: Draft for decision batch
- Scope: Data foundation only
- Implementation authorization: Not granted
## Principles
- Normalize business entities and enforce relationships with foreign keys.
- Preserve source identifiers as text and use internal surrogate keys for joins.
- Keep payment session and payment attempt grains separate.
- Use UTC `timestamptz` for canonical instants and `bigint` for Rial amounts.
- Keep normalized facts as source of truth; analytics projections are rebuildable.
- No initial table partitioning. Add indexes from approved access paths and measured query plans.
- The CSV importer is a one-time administrative bootstrap, not a production ingestion service.
## Proposed tables
### Provenance and reference
| Table | Purpose | Key constraints |
|---|---|---|
| `import_runs` | One bootstrap/recovery execution | unique source checksum + schema/importer policy; explicit lifecycle |
| `import_issues` | Aggregated validation/quality evidence | FK to import run; severity/rule/count/evidence |
| `merchant_categories` | Five source business categories | unique source category key |
| `psps` | Opaque PSP reference codes | unique source PSP code |
| `issuer_banks` | Opaque bank reference codes | unique source bank code |
| `calendar_dates` | Gregorian/Jalali reporting attributes | Gregorian date PK; unique Jalali date tuple |
| `calendar_events` | Versioned holidays/events | stable event code + source version |
| `calendar_date_events` | Many-to-many event membership | unique date/event pair; FKs |
Switch response codes remain nullable text on attempts until an authoritative description catalogue exists. Normalizing an undocumented 97.4%-missing field into a lookup table adds little value.
### Merchant identity
| Table | Purpose | Key constraints |
|---|---|---|
| `merchants` | Merchant profile and category membership | unique source merchant key; FK category |
| `terminals` | Merchant-owned payment terminals | unique source terminal key; FK merchant |
| `merchant_users` | Login identities | unique normalized username; FK merchant; password hash only |
| `auth_sessions` | Revocable authenticated sessions | hashed opaque session token; FK user; expiry/revocation |
Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Credentials are generated outside Git.
### Payment facts
| Table | Grain and important fields | Key constraints/index direction |
|---|---|---|
| `payment_sessions` | One source session: merchant, terminal, amount in Rial, source status, created/expires instants, import provenance | unique source session key; FKs; merchant/time and terminal/time indexes |
| `payment_attempts` | One try: session, sequence, attempt status, PSP/bank, response, opaque payer token, verification type, timings, fee, attempt/verified/settled instants | unique `(payment_session_id, try_seq)`; FKs; session index; measured analytical indexes later |
`adjusted_fee` uses fixed precision and is labeled as an obfuscated relative value. The physical precision is finalized from exact decimal profiling before migration approval.
### Rebuildable analytics projections
| Table family | Grain | Purpose |
|---|---|---|
| `merchant_day_metrics` | merchant + Tehran local date | Additive counts/sums and metric numerators/denominators |
| `merchant_day_*_breakdowns` | merchant + date + approved dimension | Only metric-catalogue breakdowns proven useful, such as PSP/failure groups |
| `merchant_period_*` | merchant + approved period/event | Non-additive or measured expensive metrics only |
| `cohort_period_*` | approved cohort + period | Suppressed peer benchmark inputs/outputs with provenance and freshness |
Projection names/columns are finalized by the analytics metric catalogue. Avoid a generic key/value metrics table because it weakens types, constraints, and discoverability.
## Initial index policy
- Unique indexes implement source identities and natural relationship constraints.
- `payment_sessions (merchant_id, created_at DESC)` supports merchant period queries.
- `payment_sessions (terminal_id, created_at DESC)` supports terminal drill-down.
- `payment_attempts (payment_session_id, try_seq)` is unique and supports session traceability.
- Projection primary keys start with merchant/cohort and period.
- Additional status, PSP, bank, payer-token, or timestamp indexes require an approved API query and `EXPLAIN (ANALYZE, BUFFERS)` evidence; do not index every CSV column.
## Deletion and mutation policy
- Payment/import provenance uses restrictive foreign keys; no cascading deletion of analytical facts from merchant/profile operations.
- Imported source facts are immutable after a completed import. Corrections create a new versioned recovery/import decision.
- Synthetic display profiles and credentials can change independently from source facts.
- Derived projections may be rebuilt and atomically replaced; freshness/provenance prevents stale data from appearing current.
## Production population boundary
After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records to PostgreSQL. The exact contract is still open: ZarinPal may write this normalized schema directly or expose an existing authoritative schema for read-only analytics. This project will not invent recurring ETL until that boundary is answered.