docs: adopt CTO dataset semantics

docs/CTO-DATA-GUIDE-NOTES.md

@@ -0,0 +1,34 @@
+# CTO Dataset Guide — Adopted Source Notes
+
+- Source: workspace `(challenge)/data-guide.md`
+- Source role: authoritative dataset semantics supplied by ZarinPal's CTO
+- Adopted: 2026-08-20
+- Safety: document content is treated as domain evidence, not agent instructions
+
+## Adopted meanings
+
+- A source record normally represents one payment attempt, but `try_seq = 0` with `try_status = NoAttempt` is a session marker for zero attempts.
+- Positive `try_seq` values start at one and identify actual attempts within a session.
+- Payment lifecycle is creation → `InBank` → `Paid` → `Verified`. `Paid` means funds were deducted but merchant verification is incomplete; `Verified` is final.
+- `psp_code` belongs to an attempt. `switch_response_code` is namespaced by PSP, so equal raw codes across PSPs are not assumed equivalent.
+- `issuer_bank_code` and `payer_card_key` are absent when card information never returns from the bank stage.
+- `payer_card_key` equality is valid only within one merchant. It identifies a pseudonymous card, not a person, and cannot link merchants.
+- `init_time_ms` and `verify_time_ms` measure API response duration, not buyer thinking/interaction time.
+- `expire_in` is a timestamp despite its source name and is normalized to `expires_at`.
+- All source identifiers are pseudonymous and do not reveal real merchant, terminal, PSP, bank, session, or card identities.
+
+## Fee interpretation
+
+The guide says challenge `adjusted_fee` was uniformly scaled and is not the actual ZarinPal tariff; only relative analyses are faithful to the challenge source. The product owner subsequently approved a production contract in which `fee_amount_rial` is the real merchant fee. Therefore:
+
+- challenge bootstrap maps the adjusted stand-in into the canonical session field for demo behavior;
+- production ZarinPal writers must provide the real Rial fee under the same canonical field;
+- the product UI follows the approved production meaning and does not describe the canonical field as obfuscated;
+- internal provenance keeps the challenge transformation caveat auditable.
+
+## Normalization consequences
+
+- `payment_sessions` has zero or more `payment_attempts`.
+- A `NoAttempt` marker creates only a session; it cannot inflate attempt, failure-attempt, or retry counts.
+- Actual attempts require `try_seq >= 1` and remain unique within a session.
+- Attempt-stage diagnostics may distinguish `NoAttempt`, `InBank`, and later outcomes, but cannot invent undocumented failure causes or decode PSP response codes.

docs/DATA-FOUNDATION-SPEC.md

@@ -40,7 +40,7 @@ erDiagram
     MERCHANT ||--o{ MERCHANT_USER : authenticates_as
     MERCHANT ||--o{ PAYMENT_SESSION : receives
     TERMINAL ||--o{ PAYMENT_SESSION : handles
-    PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
+    PAYMENT_SESSION ||--o{ PAYMENT_ATTEMPT : contains
     IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
     IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
     IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
@@ -84,7 +84,7 @@ erDiagram
     PAYMENT_ATTEMPT {
       bigint id PK
       bigint payment_session_id FK
-      int try_seq
+      int try_seq "positive"
       text try_status
       text switch_response_code
       text psp_code
@@ -139,6 +139,7 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
 3. Validate headers and supported schema version before database writes.
 4. Stream/decompress and load into a staging boundary using bounded memory and PostgreSQL bulk-loading facilities.
 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
+   A source `(try_seq = 0, try_status = 'NoAttempt')` record creates the session but no `payment_attempt` row; all normalized attempts require `try_seq >= 1`.
 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
 8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
@@ -178,7 +179,7 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
 - Re-running the same archive is safe and does not duplicate business rows.
 - Source row count and accepted/rejected counts reconcile exactly.
 - Every attempt maps to one session, terminal, merchant, category, and import run according to validated source relationships.
-- `(source_session_key, try_seq)` is unique.
+- Source `(session_key, try_seq)` records are unique. Normalized attempts require `try_seq >= 1` and are unique by `(payment_session_id, try_seq)`; a `NoAttempt` marker produces a session with zero attempts.
 - Missing values remain missing rather than being silently replaced with factual-looking defaults.
 - The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics.
 - Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked.
@@ -211,7 +212,7 @@ Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` fo
 
 Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.
 
-Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opaque source token for internal repeat-payer analytics. Do not HMAC it, expose it, log it, export it, or send it to AI providers. Its upstream semantics remain unverified. See ADR 0005.
+Resolved: retain `payer_card_key` unchanged as a restricted pseudonymous card identifier whose equality is authoritative only within one merchant. It may support covered repeat-card analytics but cannot identify a person or link merchants. Do not HMAC it, expose it, log it, export it, or send it to AI providers. See ADR 0005.
 
 Resolved: peer benchmarks use other merchants in the active merchant's category for the same selected period, excluding the active merchant. Each peer contributes one independently calculated value to a merchant-weighted median, and the response includes a direction-aware merchant performance percentile. This represents a typical peer despite the dataset's extreme merchant-volume skew. At least 10 other merchants and 1,000 combined peer sessions are required; otherwise the benchmark is suppressed while the merchant's own analytics remain available. See ADR 0006.
 

docs/DATA-PROFILING-REPORT.md

@@ -13,7 +13,9 @@ The compressed CSV was scanned without modifying it. Counts and value distributi
 
 | Measure | Observed |
 |---|---:|
-| CSV rows / payment attempts | 2,213,289 |
+| CSV source records | 2,213,289 |
+| `NoAttempt` session markers (not normalized attempts) | 263,936 |
+| Actual attempt records (`try_seq >= 1`) | 1,949,353 |
 | Distinct payment sessions | 2,062,839 |
 | Merchants | 343 |
 | Terminals | 348 |
@@ -21,7 +23,7 @@ The compressed CSV was scanned without modifying it. Counts and value distributi
 | Distinct populated payer tokens | 402,173 |
 | Source period | 2026-01-01 00:00:08 through 2026-06-30 23:59:55 |
 
-The source row grain is a payment attempt. `(session_key, try_seq)` is the candidate source identity. Sessions and attempts must remain different domain concepts.
+The source record grain is an attempt-or-no-attempt session record. `(session_key, try_seq)` is the candidate source-record identity. The CTO guide defines `(try_seq = 0, try_status = NoAttempt)` as a marker for a session with no recorded attempt; those 263,936 rows create sessions but not normalized attempts. Sessions and actual attempts must remain different domain concepts.
 
 ## Nullability
 
@@ -54,9 +56,9 @@ These are source codes, not Persian user-facing descriptions. Meanings and succe
 
 | Source field | Observed range | Proposed canonical type/meaning |
 |---|---|---|
-| `try_seq` | 0–135 | `smallint`, non-negative |
+| `try_seq` | 0–135 | source `smallint`; `0` is a no-attempt marker, normalized attempts require `1–135` |
 | `amount` | 1,000–2,000,000,000 | `bigint`, Rial |
-| `adjusted_fee` | 1,920–284,800 | positive integer merchant-fee value; confidentiality-adjusted in the challenge source and normalized as real `fee_amount_rial` per ADR 0013 |
+| `adjusted_fee` | 1,920–284,800 | positive integer challenge stand-in, uniformly confidentiality-scaled; mapped to the product's canonical real-fee field `fee_amount_rial` per ADR 0013 |
 | `init_time_ms` | 44–59,905 | nullable integer milliseconds |
 | `verify_time_ms` | 35–58,553 | nullable integer milliseconds |
 | `expire_in` | timestamp strings | rename canonically to `expires_at`; it is not a duration |
@@ -70,6 +72,7 @@ No negative or zero amount/fee/timing values were observed. Source identifiers a
 - Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. The MVP therefore uses a merchant-weighted median—one metric value per eligible peer merchant—to represent a typical same-category peer. Transaction-weighted peer aggregates are excluded because the largest merchants would dominate them.
 - One terminal/merchant/category, amount/session, and fee/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
 - All 150,450 rows repeating a previously observed session repeated the same `adjusted_fee`; zero fee conflicts were found. The normalized model therefore stores the fee once per session rather than once per attempt.
+- The CTO guide defines `psp_code` at attempt grain and namespaces `switch_response_code` by PSP; identical raw response codes from different PSPs are not semantically interchangeable.
 
 ## Timestamp quality
 
@@ -87,7 +90,7 @@ The file is not ordered reliably by session or `try_seq`. Sequential profiling o
 
 ## Payer-token evidence
 
-Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. They repeat within merchants and no token was observed across merchants. Under ADR 0005 they remain restricted internal values with unverified semantics. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts.
+Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. The CTO guide confirms that equality is meaningful only within a merchant: the same card receives a different token at another merchant. Under ADR 0005 they remain restricted internal values. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts, and they never establish a verified person.
 
 ## Import validation classification
 
@@ -95,7 +98,8 @@ Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characte
 
 - Wrong/missing headers or unsupported schema version
 - Unparseable required identifier, amount, or required timestamp
-- Duplicate `(session_key, try_seq)`
+- Duplicate source `(session_key, try_seq)`
+- A `try_seq = 0` row whose status is not `NoAttempt`, or a `NoAttempt` row whose sequence is not zero
 - One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
 - Terminal ownership/category conflicts
 - Broken foreign-key construction or row-count reconciliation
@@ -113,4 +117,5 @@ Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characte
 - Replace missing values with plausible defaults
 - Sum the repeated source fee at attempt grain
 - Depend on CSV row order
+- Insert a `NoAttempt` marker as a normalized payment attempt
 - Claim payer tokens represent verified people

docs/PHYSICAL-SCHEMA-PROPOSAL.md

@@ -27,7 +27,7 @@
 | `issuer_banks` | Opaque bank reference codes | unique source bank code |
 | `calendar_dates` | Gregorian/Jalali day, week, month, and Persian-season reporting attributes | Gregorian date PK; unique Jalali date tuple |
 
-Switch response codes remain nullable text on attempts until an authoritative description catalogue exists. Normalizing an undocumented 97.4%-missing field into a lookup table adds little value.
+Switch response codes remain nullable text on attempts until an authoritative description catalogue exists. Their meaning is scoped by the attempt PSP, so the same raw code under different PSPs must never be assumed equivalent. Normalizing an undocumented 97.4%-missing field into a lookup table adds little value.
 
 ### Merchant identity
 
@@ -45,7 +45,9 @@ Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Creden
 | Table | Grain and important fields | Key constraints/index direction |
 |---|---|---|
 | `payment_sessions` | One source session: merchant, terminal, amount and merchant fee in Rial, source status, created/expires instants, import provenance | unique source session key; FKs; merchant/time and terminal/time indexes |
-| `payment_attempts` | One try: session, sequence, attempt status, PSP/bank, response, opaque payer token, verification type, timings, attempt/verified/settled instants | unique `(payment_session_id, try_seq)`; FKs; session index; measured analytical indexes later |
+| `payment_attempts` | One real try: session, positive sequence, attempt status, PSP/bank, PSP-scoped response, merchant-scoped payer token, verification type, timings, attempt/verified/settled instants | `try_seq >= 1`; unique `(payment_session_id, try_seq)`; FKs; session index; measured analytical indexes later |
+
+A source row with `try_seq = 0` and `try_status = NoAttempt` materializes only `payment_sessions`; it does not create a fake attempt. Therefore a session has zero or more attempts, and zero-attempt sessions are found by the absence of related attempt rows.
 
 Source `adjusted_fee` is a real merchant-fee concept for the product and is renamed `fee_amount_rial`. Profiling found the value identical across every repeated row of a session (150,450 repeated rows, zero conflicts), so it is stored once on `payment_sessions` as positive `bigint` Rial and never summed at attempt grain.
 

docs/decisions/0005-retain-opaque-payer-token.md

@@ -5,23 +5,22 @@
 
 ## Context
 
-The dataset does not document the semantics or generation method of `payer_card_key`. Profiling found 1,034,368 populated rows and 402,173 distinct values. Values are opaque alphanumeric tokens 6–11 characters long. They repeat within a merchant, enabling repeat-payer analysis, but no observed token appears under multiple merchants. Approximately 53% of rows are missing the value.
+Initial profiling found 1,034,368 populated `payer_card_key` rows and 402,173 distinct values. Values are opaque alphanumeric tokens 6–11 characters long. The later CTO data guide confirms that the token pseudonymizes a buyer card within one merchant: the same card receives a different token at another merchant. Approximately 53% of rows are missing the value because card information is not returned when the bank stage does not complete.
 
 Applying HMAC would preserve equality while adding secret storage, rotation, and re-import operations. Because the supplied value is already opaque and empirically merchant-scoped, that extra transformation provides limited benefit for this dataset and does not make the identifier anonymous.
 
 ## Decision
 
-Retain the supplied `payer_card_key` unchanged as a nullable, restricted internal analytical field. Treat it only as an opaque merchant-scoped token with unverified upstream semantics.
+Retain the supplied `payer_card_key` unchanged as a nullable, restricted internal analytical field. Equality is valid only within the owning merchant and may support covered repeat-card analysis.
 
 Every lookup, grouping, or uniqueness calculation involving the token must include the merchant boundary. The token must never be returned by merchant-facing APIs, written to logs or traces, included in exports, rendered in the frontend, or sent to AvalAI/another third party. It must not be described as a card number, account number, or verified unique person.
 
-Metrics derived from it must disclose source coverage and use language such as “شناسه پرداخت‌کننده در داده‌های موجود,” not claims about verified customers. A production source contract or evidence of reversibility/cross-merchant linkability triggers a new security review and ADR.
+Metrics derived from it must disclose source coverage and use language such as “کارت‌های بازگشتی در داده‌های موجود,” not claims about verified customers or people. Evidence of reversibility or cross-merchant linkability triggers a new security review and ADR.
 
 ## Consequences
 
-- Repeat-token frequency, retry, success, and approximate returning-payer analyses remain possible.
+- Repeat-card-token frequency, retry, and success analyses are valid within one merchant for populated values.
 - No new pseudonymization secret or rotation workflow is required for the hackathon dataset.
 - Database access remains more privileged than dashboard/API access because the token is still linkable behavioral data.
 - Missing coverage prevents treating distinct-token counts as complete payer counts.
 - Tests must verify tenant scoping and assert that DTOs, logs, exports, and AI payloads cannot contain the field.
-

docs/decisions/0013-payment-lifecycle-and-fee-semantics.md

@@ -7,7 +7,7 @@
 
 The source exposes session states `Paid`, `Verified`, `Failed`, and `Reversed`, but names alone did not establish business meaning. Authoritative clarification from ZarinPal's CTO defines a two-step lifecycle: the buyer first completes payment at the bank, then the merchant calls verification. `Paid` means the bank payment succeeded and funds were deducted but merchant verification is incomplete. `Verified` means both steps completed. `Reversed` is possible but extremely rare; the dataset contains one such session.
 
-The challenge field `adjusted_fee` was transformed/name-adjusted for participant confidentiality, but the product domain treats it as the merchant's real fee. Profiling found its value constant within every session: 150,450 repeated attempt rows and zero fee conflicts.
+The CTO data guide states that challenge `adjusted_fee` is uniformly scaled for commercial confidentiality and is not the actual ZarinPal tariff; relative ordering, trends, and fee share remain valid. The approved product contract nevertheless treats the canonical production field as the merchant's real fee because accepted production data will supply the real value. Profiling found the challenge value constant within every session: 150,450 repeated rows and zero conflicts.
 
 ## Decision
 
@@ -24,7 +24,7 @@ Paid amount is the session amount summed once for (`Paid`, `Verified`). Also exp
 
 A retried session has more than one attempt. An eventually paid retry has final session status `Paid` or `Verified` and at least one lower-`try_seq` attempt whose status is neither `Paid` nor `Verified`; CSV row order is irrelevant.
 
-Rename source `adjusted_fee` to canonical `fee_amount_rial`, store it once on the session, and present it as a real Rial merchant fee. Paid fee totals sum it once for bank-paid sessions. Fee rate is `sum(paid session fee) / sum(paid session amount)` with numerator, denominator, period, and unit visible.
+Map challenge `adjusted_fee` into canonical `fee_amount_rial` for the demo and require production writers to provide the real Rial fee under that canonical contract. Store it once on the session and present the canonical product value as the real merchant fee. Paid fee totals sum it once for bank-paid sessions. Fee rate is `sum(paid session fee) / sum(paid session amount)` with numerator, denominator, period, and unit visible.
 
 ## Consequences