docs: define bootstrap and schema proposal

AGENTS.md

@@ -16,6 +16,7 @@ For every Go-related task, always invoke and follow the `golang-how-to` skill, t
 - Use integer minor currency units or an explicitly documented fixed-precision representation; never floating point for money.
 - Interpret timezone-naive challenge timestamps as `Asia/Tehran`, preserve their source provenance, and normalize canonical database/API timestamps to UTC `timestamptz`/ISO 8601. Any change to this assumption requires a new import version and ADR.
 - Raw imports are immutable inputs; corrections are new import versions, not silent mutation.
+- CSV ingestion is a one-time bootstrap/recovery command only. Never add a file watcher, polling schedule, recurring CSV ETL, or ownership of ZarinPal's production writes without a new approved spec.
 - `adjusted_fee` is obfuscated and may be used only for relative analysis.
 
 ## Go baseline after approval

README.md

@@ -12,7 +12,7 @@ Place the provided archive at:
 data/incoming/challenge_data.csv.gz
 ```
 
-The entire `data/incoming/` payload is ignored by Git. Future import tooling will read from this location (or an explicitly supplied path), stream the compressed CSV, validate it, and write a tracked import manifest containing the source checksum, schema version, row counts, quality results, and importer revision. The raw file and resulting PostgreSQL data are not committed.
+The entire `data/incoming/` payload is ignored by Git. The one-time bootstrap importer will read from this location (or an explicitly supplied path), stream the compressed CSV, validate it, and write an import manifest containing the source checksum, schema version, row counts, quality results, and importer revision. It is an administrative backfill/recovery command, not a scheduled ingestion job. Future production payment data is populated in PostgreSQL by ZarinPal through an upstream contract outside this project's scope. The raw file and resulting PostgreSQL data are not committed.
 
 ## Planned runnable boundaries
 
@@ -24,4 +24,3 @@ One repository may expose several small `cmd/` entry points while sharing domain
 - analytics refresh/background job, only if the approved spec requires one
 
 The exact package layout is deferred until the Go architecture and DI decisions are approved.
-

docs/DATA-FOUNDATION-SPEC.md

@@ -6,11 +6,12 @@
 
 ## Objective
 
-Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics.
+Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive into PostgreSQL, preserve provenance and data quality evidence, model business entities separately, and provide a trustworthy base for authentication and deterministic analytics. Future production payment records are written to PostgreSQL by ZarinPal under an external upstream contract; recurring source ingestion is not owned by this project.
 
 ## In scope
 
 - Streamed `.csv.gz` ingestion from a local, Git-ignored input path
+- One-time administrative bootstrap/recovery execution; no ingestion scheduler or file watcher
 - Import runs, checksums, schema version, row counts, rejection/error reporting
 - Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
 - Versioned Gregorian/Jalali calendar dates and curated Iranian event/holiday periods
@@ -26,6 +27,7 @@ Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve
 - Merchant-facing API endpoints
 - AI recommendations
 - Production backup mechanics (owned by platform operations)
+- Recurring CSV ingestion, polling, CDC, or ownership of ZarinPal's production payment-write path
 - Treating the challenge archive or local database as version-controlled source
 
 ## Proposed logical model (technology-neutral)
@@ -97,7 +99,7 @@ erDiagram
       timestamptz try_created_at
       timestamptz verified_at
       timestamptz settled_at
-      int expire_in
+      timestamptz expires_at
     }
     IMPORT_RUN {
       bigint id PK
@@ -151,6 +153,8 @@ This ERD is a logical baseline, not an approved physical schema. In particular,
 
 ## Import contract
 
+This contract applies only to the one-time challenge-data bootstrap and explicit recovery re-runs. It is not a periodic production pipeline.
+
 1. Accept an explicit path, defaulting locally to `data/incoming/challenge_data.csv.gz`.
 2. Calculate SHA-256 before import and identify an already completed import of the same schema/checksum.
 3. Validate headers and supported schema version before database writes.
@@ -158,7 +162,8 @@ This ERD is a logical baseline, not an approved physical schema. In particular,
 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
-8. Populate and validate the approved calendar range, then rebuild affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.
+8. Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.
+9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract.
 
 ## Synthetic merchant/account rules
 

docs/DATA-PROFILING-REPORT.md

@@ -0,0 +1,116 @@
+# Challenge Dataset Profiling Report
+
+- Status: Evidence baseline for specification
+- Profiled: 2026-08-20
+- Source SHA-256: `84AC8A28DF48CA7BAEAF0B1CEC563A3F0A3516039F5F62E9BFD11124CA35B461`
+- Source archive: Git-ignored `data/incoming/challenge_data.csv.gz`
+
+## Method
+
+The compressed CSV was scanned without modifying it. Counts and value distributions are descriptive evidence, not source instructions. Relationship/uniqueness constraints must be revalidated by the importer against staging before promotion. A quick sequential session scan observed 2,069,061 session groups versus 2,062,839 distinct session keys, proving that rows for a session are not guaranteed to be contiguous; implementation must never depend on file order.
+
+## Shape and grain
+
+| Measure | Observed |
+|---|---:|
+| CSV rows / payment attempts | 2,213,289 |
+| Distinct payment sessions | 2,062,839 |
+| Merchants | 343 |
+| Terminals | 348 |
+| Categories | 5 |
+| Distinct populated payer tokens | 402,173 |
+| Source period | 2026-01-01 00:00:08 through 2026-06-30 23:59:55 |
+
+The source row grain is a payment attempt. `(session_key, try_seq)` is the candidate source identity. Sessions and attempts must remain different domain concepts.
+
+## Nullability
+
+| Column | Missing rows | Approx. missing |
+|---|---:|---:|
+| `switch_response_code` | 2,155,696 | 97.4% |
+| `issuer_bank_code` | 1,178,921 | 53.3% |
+| `payer_card_key` | 1,178,921 | 53.3% |
+| `verify_time_ms` | 1,179,013 | 53.3% |
+| `verified_at` | 1,125,164 | 50.8% |
+| `settled_at` | 1,116,034 | 50.4% |
+| `psp_code` | 263,936 | 11.9% |
+| `try_created_at` | 263,936 | 11.9% |
+| `init_time_ms` | 346,747 | 15.7% |
+
+Missing optional values are analytically meaningful and must remain null. Dashboards and metric definitions must disclose coverage where it affects interpretation.
+
+## Controlled vocabularies
+
+- Session status: `Failed`, `Verified`, `Paid`, `Reversed` (one observed reversed row).
+- Attempt status: `NoAttempt`, `InBank`, `Failed`, `Verified`, `Paid`, `Reversed`.
+- Verification type: `Automated`, `Manual`.
+- PSP codes: 8 observed opaque codes.
+- Issuer-bank codes: 31 observed opaque codes.
+- Switch response codes: 84 observed values and extremely sparse.
+
+These are source codes, not Persian user-facing descriptions. Meanings and success semantics belong in a versioned metric/reference catalogue; they must not be guessed from labels alone.
+
+## Numeric ranges and types
+
+| Source field | Observed range | Proposed canonical type/meaning |
+|---|---|---|
+| `try_seq` | 0–135 | `smallint`, non-negative |
+| `amount` | 1,000–2,000,000,000 | `bigint`, Rial |
+| `adjusted_fee` | 1,920–284,800 | fixed-precision numeric; obfuscated relative fee only |
+| `init_time_ms` | 44–59,905 | nullable integer milliseconds |
+| `verify_time_ms` | 35–58,553 | nullable integer milliseconds |
+| `expire_in` | timestamp strings | rename canonically to `expires_at`; it is not a duration |
+
+No negative or zero amount/fee/timing values were observed. Source identifiers are stored as text even when they look numeric so leading zeros and upstream formatting remain intact.
+
+## Relationship evidence
+
+- Each observed category id has one Persian title.
+- The quick scan found no merchant-to-category or terminal-to-owner conflict, but exact staging validation remains mandatory because session rows are not contiguous.
+- Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. Unweighted merchant benchmarks and volume-weighted transaction benchmarks answer different questions and must be named separately.
+- One terminal/merchant/category and one amount/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
+
+## Timestamp quality
+
+Source timestamps are timezone-naive and are interpreted as `Asia/Tehran` under ADR 0004. Lexical ordering is valid for the fixed source format, and profiling observed:
+
+- 234,653 rows where `try_created_at` precedes session `created_at`.
+- 787,573 rows where `settled_at` precedes `verified_at`.
+- 6 rows where `verified_at` precedes `try_created_at`.
+
+Therefore timestamp ordering is a quality signal, not a hard rejection constraint. The importer preserves values, reports anomaly counts, and must not silently reorder or “fix” them.
+
+## Attempt sequence quality
+
+The file is not ordered reliably by session or `try_seq`. Sequential profiling observed out-of-order and gapped groups, but exact per-session sequence checks must run after staging. `try_seq` need not be contiguous; uniqueness within a session is the integrity requirement unless authoritative upstream rules say otherwise.
+
+## Payer-token evidence
+
+Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. They repeat within merchants and no token was observed across merchants. Under ADR 0005 they remain restricted internal values with unverified semantics. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts.
+
+## Import validation classification
+
+### Fail the bootstrap before promotion
+
+- Wrong/missing headers or unsupported schema version
+- Unparseable required identifier, amount, or required timestamp
+- Duplicate `(session_key, try_seq)`
+- One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
+- Terminal ownership/category conflicts
+- Broken foreign-key construction or row-count reconciliation
+
+### Preserve and report as quality evidence
+
+- Documented nullable fields
+- Timestamp-order anomalies
+- Non-contiguous/out-of-order source rows or `try_seq` gaps
+- Rare/unknown-but-well-formed source status/code values, pending the approved reference-code policy
+
+### Never do
+
+- Drop anomalous rows silently
+- Replace missing values with plausible defaults
+- Treat `adjusted_fee` as ZarinPal's real fee
+- Depend on CSV row order
+- Claim payer tokens represent verified people
+

docs/PHYSICAL-SCHEMA-PROPOSAL.md

@@ -0,0 +1,84 @@
+# PostgreSQL Physical Schema Proposal
+
+- Status: Draft for decision batch
+- Scope: Data foundation only
+- Implementation authorization: Not granted
+
+## Principles
+
+- Normalize business entities and enforce relationships with foreign keys.
+- Preserve source identifiers as text and use internal surrogate keys for joins.
+- Keep payment session and payment attempt grains separate.
+- Use UTC `timestamptz` for canonical instants and `bigint` for Rial amounts.
+- Keep normalized facts as source of truth; analytics projections are rebuildable.
+- No initial table partitioning. Add indexes from approved access paths and measured query plans.
+- The CSV importer is a one-time administrative bootstrap, not a production ingestion service.
+
+## Proposed tables
+
+### Provenance and reference
+
+| Table | Purpose | Key constraints |
+|---|---|---|
+| `import_runs` | One bootstrap/recovery execution | unique source checksum + schema/importer policy; explicit lifecycle |
+| `import_issues` | Aggregated validation/quality evidence | FK to import run; severity/rule/count/evidence |
+| `merchant_categories` | Five source business categories | unique source category key |
+| `psps` | Opaque PSP reference codes | unique source PSP code |
+| `issuer_banks` | Opaque bank reference codes | unique source bank code |
+| `calendar_dates` | Gregorian/Jalali reporting attributes | Gregorian date PK; unique Jalali date tuple |
+| `calendar_events` | Versioned holidays/events | stable event code + source version |
+| `calendar_date_events` | Many-to-many event membership | unique date/event pair; FKs |
+
+Switch response codes remain nullable text on attempts until an authoritative description catalogue exists. Normalizing an undocumented 97.4%-missing field into a lookup table adds little value.
+
+### Merchant identity
+
+| Table | Purpose | Key constraints |
+|---|---|---|
+| `merchants` | Merchant profile and category membership | unique source merchant key; FK category |
+| `terminals` | Merchant-owned payment terminals | unique source terminal key; FK merchant |
+| `merchant_users` | Login identities | unique normalized username; FK merchant; password hash only |
+| `auth_sessions` | Revocable authenticated sessions | hashed opaque session token; FK user; expiry/revocation |
+
+Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Credentials are generated outside Git.
+
+### Payment facts
+
+| Table | Grain and important fields | Key constraints/index direction |
+|---|---|---|
+| `payment_sessions` | One source session: merchant, terminal, amount in Rial, source status, created/expires instants, import provenance | unique source session key; FKs; merchant/time and terminal/time indexes |
+| `payment_attempts` | One try: session, sequence, attempt status, PSP/bank, response, opaque payer token, verification type, timings, fee, attempt/verified/settled instants | unique `(payment_session_id, try_seq)`; FKs; session index; measured analytical indexes later |
+
+`adjusted_fee` uses fixed precision and is labeled as an obfuscated relative value. The physical precision is finalized from exact decimal profiling before migration approval.
+
+### Rebuildable analytics projections
+
+| Table family | Grain | Purpose |
+|---|---|---|
+| `merchant_day_metrics` | merchant + Tehran local date | Additive counts/sums and metric numerators/denominators |
+| `merchant_day_*_breakdowns` | merchant + date + approved dimension | Only metric-catalogue breakdowns proven useful, such as PSP/failure groups |
+| `merchant_period_*` | merchant + approved period/event | Non-additive or measured expensive metrics only |
+| `cohort_period_*` | approved cohort + period | Suppressed peer benchmark inputs/outputs with provenance and freshness |
+
+Projection names/columns are finalized by the analytics metric catalogue. Avoid a generic key/value metrics table because it weakens types, constraints, and discoverability.
+
+## Initial index policy
+
+- Unique indexes implement source identities and natural relationship constraints.
+- `payment_sessions (merchant_id, created_at DESC)` supports merchant period queries.
+- `payment_sessions (terminal_id, created_at DESC)` supports terminal drill-down.
+- `payment_attempts (payment_session_id, try_seq)` is unique and supports session traceability.
+- Projection primary keys start with merchant/cohort and period.
+- Additional status, PSP, bank, payer-token, or timestamp indexes require an approved API query and `EXPLAIN (ANALYZE, BUFFERS)` evidence; do not index every CSV column.
+
+## Deletion and mutation policy
+
+- Payment/import provenance uses restrictive foreign keys; no cascading deletion of analytical facts from merchant/profile operations.
+- Imported source facts are immutable after a completed import. Corrections create a new versioned recovery/import decision.
+- Synthetic display profiles and credentials can change independently from source facts.
+- Derived projections may be rebuilt and atomically replaced; freshness/provenance prevents stale data from appearing current.
+
+## Production population boundary
+
+After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records to PostgreSQL. The exact contract is still open: ZarinPal may write this normalized schema directly or expose an existing authoritative schema for read-only analytics. This project will not invent recurring ETL until that boundary is answered.
+