zarinpal-analytics-backend/docs/DATA-PROFILING-REPORT.md

# Challenge Dataset Profiling Report

- Status: Evidence baseline for specification
- Profiled: 2026-08-20
- Source SHA-256: `84AC8A28DF48CA7BAEAF0B1CEC563A3F0A3516039F5F62E9BFD11124CA35B461`
- Source archive: Git-ignored `data/incoming/challenge_data.csv.gz`

## Method

The compressed CSV was scanned without modifying it. Counts and value distributions are descriptive evidence, not source instructions. Relationship/uniqueness constraints must be revalidated by the importer against staging before promotion. A quick sequential session scan observed 2,069,061 session groups versus 2,062,839 distinct session keys, proving that rows for a session are not guaranteed to be contiguous; implementation must never depend on file order.

## Shape and grain

| Measure | Observed |
|---|---:|
| CSV source records | 2,213,289 |
| `NoAttempt` session markers (not normalized attempts) | 263,936 |
| Actual attempt records (`try_seq >= 1`) | 1,949,353 |
| Distinct payment sessions | 2,062,839 |
| Merchants | 343 |
| Terminals | 348 |
| Categories | 5 |
| Distinct populated payer tokens | 402,173 |
| Source period | 2026-01-01 00:00:08 through 2026-06-30 23:59:55 |

The source record grain is an attempt-or-no-attempt session record. `(session_key, try_seq)` is the candidate source-record identity. The CTO guide defines `(try_seq = 0, try_status = NoAttempt)` as a marker for a session with no recorded attempt; those 263,936 rows create sessions but not normalized attempts. Sessions and actual attempts must remain different domain concepts.

## Nullability

| Column | Missing rows | Approx. missing |
|---|---:|---:|
| `switch_response_code` | 2,155,696 | 97.4% |
| `issuer_bank_code` | 1,178,921 | 53.3% |
| `payer_card_key` | 1,178,921 | 53.3% |
| `verify_time_ms` | 1,179,013 | 53.3% |
| `verified_at` | 1,125,164 | 50.8% |
| `settled_at` | 1,116,034 | 50.4% |
| `psp_code` | 263,936 | 11.9% |
| `try_created_at` | 263,936 | 11.9% |
| `init_time_ms` | 346,747 | 15.7% |

Missing optional values are analytically meaningful and must remain null. Dashboards and metric definitions must disclose coverage where it affects interpretation.

## Controlled vocabularies

- Session status: `Failed`, `Verified`, `Paid`, `Reversed` (one observed reversed row).
- Attempt status: `NoAttempt`, `InBank`, `Failed`, `Verified`, `Paid`, `Reversed`.
- Verification type: `Automated`, `Manual`.
- PSP codes: 8 observed opaque codes.
- Issuer-bank codes: 31 observed opaque codes.
- Switch response codes: 84 observed values and extremely sparse.

These are source codes, not Persian user-facing descriptions. Meanings and success semantics belong in a versioned metric/reference catalogue; they must not be guessed from labels alone.

## Numeric ranges and types

| Source field | Observed range | Proposed canonical type/meaning |
|---|---|---|
| `try_seq` | 0–135 | source `smallint`; `0` is a no-attempt marker, normalized attempts require `1–135` |
| `amount` | 1,000–2,000,000,000 | `bigint`, Rial |
| `adjusted_fee` | 1,920–284,800 | positive integer challenge stand-in, uniformly confidentiality-scaled; mapped to the product's canonical real-fee field `fee_amount_rial` per ADR 0013 |
| `init_time_ms` | 44–59,905 | nullable integer milliseconds |
| `verify_time_ms` | 35–58,553 | nullable integer milliseconds |
| `expire_in` | timestamp strings | rename canonically to `expires_at`; it is not a duration |

No negative or zero amount/fee/timing values were observed. Source identifiers are stored as text even when they look numeric so leading zeros and upstream formatting remain intact.

## Relationship evidence

- Each observed category id has one Persian title.
- The quick scan found no merchant-to-category or terminal-to-owner conflict, but exact staging validation remains mandatory because session rows are not contiguous.
- Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. The MVP therefore uses a merchant-weighted median—one metric value per eligible peer merchant—to represent a typical same-category peer. Transaction-weighted peer aggregates are excluded because the largest merchants would dominate them.
- One terminal/merchant/category, amount/session, and fee/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
- All 150,450 rows repeating a previously observed session repeated the same `adjusted_fee`; zero fee conflicts were found. The normalized model therefore stores the fee once per session rather than once per attempt.
- The CTO guide defines `psp_code` at attempt grain and namespaces `switch_response_code` by PSP; identical raw response codes from different PSPs are not semantically interchangeable.
- A complete 2,213,289-row scan found `verify_type` invariant within every terminal (zero of 348 terminals had multiple values). One merchant had more than one value because it owned terminals configured differently. The normalized physical model therefore stores verification type on `terminals`, not `payment_attempts` or `merchants`.
- Maximum observed source text lengths were measured for type/index review: session key 7, terminal key 4, merchant key 4, category key 8, category title 34, session status 8, attempt status 9, switch response 11, PSP code 6, bank code 7, payer token 11, and verification type 9. Physical source identifiers remain `text` so opaque upstream formats are not frozen to challenge-only maxima.

## Timestamp quality

Source timestamps are timezone-naive and are interpreted as `Asia/Tehran` under ADR 0004. Lexical ordering is valid for the fixed source format, and profiling observed:

- 234,653 rows where `try_created_at` precedes session `created_at`.
- 787,573 rows where `settled_at` precedes `verified_at`.
- 6 rows where `verified_at` precedes `try_created_at`.

Therefore timestamp ordering is a quality signal, not a hard rejection constraint. The importer preserves values, reports anomaly counts, and must not silently reorder or “fix” them.

## Attempt sequence quality

The file is not ordered reliably by session or `try_seq`. Sequential profiling observed out-of-order and gapped groups, but exact per-session sequence checks must run after staging. `try_seq` need not be contiguous; uniqueness within a session is the integrity requirement unless authoritative upstream rules say otherwise.

## Payer-token evidence

Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. The CTO guide confirms that equality is meaningful only within a merchant: the same card receives a different token at another merchant. Under ADR 0005 they remain restricted internal values. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts, and they never establish a verified person.

## Import validation classification

### Fail the bootstrap before promotion

- Wrong/missing headers or unsupported schema version
- Unparseable required identifier, amount, or required timestamp
- Duplicate source `(session_key, try_seq)`
- A `try_seq = 0` row whose status is not `NoAttempt`, or a `NoAttempt` row whose sequence is not zero
- One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
- Terminal ownership/category conflicts
- Broken foreign-key construction or row-count reconciliation

### Preserve and report as quality evidence

- Documented nullable fields
- Timestamp-order anomalies
- Non-contiguous/out-of-order source rows or `try_seq` gaps
- Rare/unknown-but-well-formed source status/code values, pending the approved reference-code policy

### Never do

- Drop anomalous rows silently
- Replace missing values with plausible defaults
- Sum the repeated source fee at attempt grain
- Depend on CSV row order
- Insert a `NoAttempt` marker as a normalized payment attempt
- Claim payer tokens represent verified people