# Challenge Dataset Profiling Report

- Status: Evidence baseline for specification
- Profiled: 2026-08-20
- Source SHA-256: `84AC8A28DF48CA7BAEAF0B1CEC563A3F0A3516039F5F62E9BFD11124CA35B461`
- Source archive: Git-ignored `data/incoming/challenge_data.csv.gz`

## Method

The compressed CSV was scanned without modifying it. Counts and value distributions are descriptive evidence, not source instructions. Relationship/uniqueness constraints must be revalidated by the importer against staging before promotion. A quick sequential session scan observed 2,069,061 session groups versus 2,062,839 distinct session keys, proving that rows for a session are not guaranteed to be contiguous; implementation must never depend on file order.

## Shape and grain

| Measure | Observed |
|---|---:|
| CSV source records | 2,213,289 |
| `NoAttempt` session markers (not normalized attempts) | 263,936 |
| Actual attempt records (`try_seq >= 1`) | 1,949,353 |
| Distinct payment sessions | 2,062,839 |
| Merchants | 343 |
| Terminals | 348 |
| Categories | 5 |
| Distinct populated payer tokens | 402,173 |
| Source period | 2026-01-01 00:00:08 through 2026-06-30 23:59:55 |

The source record grain is an attempt-or-no-attempt session record. `(session_key, try_seq)` is the candidate source-record identity. The CTO guide defines `(try_seq = 0, try_status = NoAttempt)` as a marker for a session with no recorded attempt; those 263,936 rows create sessions but not normalized attempts. Sessions and actual attempts must remain different domain concepts.

## Nullability

| Column | Missing rows | Approx. missing |
|---|---:|---:|
| `switch_response_code` | 2,155,696 | 97.4% |
| `issuer_bank_code` | 1,178,921 | 53.3% |
| `payer_card_key` | 1,178,921 | 53.3% |
| `verify_time_ms` | 1,179,013 | 53.3% |
| `verified_at` | 1,125,164 | 50.8% |
| `settled_at` | 1,116,034 | 50.4% |
| `psp_code` | 263,936 | 11.9% |
| `try_created_at` | 263,936 | 11.9% |
| `init_time_ms` | 346,747 | 15.7% |

Missing optional values are analytically meaningful and must remain null. Dashboards and metric definitions must disclose coverage where it affects interpretation.

## Controlled vocabularies

- Session status: `Failed`, `Verified`, `Paid`, `Reversed` (one observed reversed row).
- Attempt status: `NoAttempt`, `InBank`, `Failed`, `Verified`, `Paid`, `Reversed`.
- Verification type: `Automated`, `Manual`.
- PSP codes: 8 observed opaque codes.
- Issuer-bank codes: 31 observed opaque codes.
- Switch response codes: 84 observed values and extremely sparse.

These are source codes, not Persian user-facing descriptions. Meanings and success semantics belong in a versioned metric/reference catalogue; they must not be guessed from labels alone.

## Numeric ranges and types

| Source field | Observed range | Proposed canonical type/meaning |
|---|---|---|
| `try_seq` | 0–135 | source `smallint`; `0` is a no-attempt marker, normalized attempts require `1–135` |
| `amount` | 1,000–2,000,000,000 | `bigint`, Rial |
| `adjusted_fee` | 1,920–284,800 | positive integer challenge stand-in, uniformly confidentiality-scaled; mapped to the product's canonical real-fee field `fee_amount_rial` per ADR 0013 |
| `init_time_ms` | 44–59,905 | nullable integer milliseconds |
| `verify_time_ms` | 35–58,553 | nullable integer milliseconds |
| `expire_in` | timestamp strings | rename canonically to `expires_at`; it is not a duration |

No negative or zero amount/fee/timing values were observed. Source identifiers are stored as text even when they look numeric so leading zeros and upstream formatting remain intact.

## Relationship evidence

- Each observed category id has one Persian title.
- The quick scan found no merchant-to-category or terminal-to-owner conflict, but exact staging validation remains mandatory because session rows are not contiguous.
- Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. The MVP therefore uses a merchant-weighted median—one metric value per eligible peer merchant—to represent a typical same-category peer. Transaction-weighted peer aggregates are excluded because the largest merchants would dominate them.
- One terminal/merchant/category, amount/session, and fee/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
- All 150,450 rows repeating a previously observed session repeated the same `adjusted_fee`; zero fee conflicts were found. The normalized model therefore stores the fee once per session rather than once per attempt.
- The CTO guide defines `psp_code` at attempt grain and namespaces `switch_response_code` by PSP; identical raw response codes from different PSPs are not semantically interchangeable.

## Timestamp quality

Source timestamps are timezone-naive and are interpreted as `Asia/Tehran` under ADR 0004. Lexical ordering is valid for the fixed source format, and profiling observed:

- 234,653 rows where `try_created_at` precedes session `created_at`.
- 787,573 rows where `settled_at` precedes `verified_at`.
- 6 rows where `verified_at` precedes `try_created_at`.

Therefore timestamp ordering is a quality signal, not a hard rejection constraint. The importer preserves values, reports anomaly counts, and must not silently reorder or “fix” them.

## Attempt sequence quality

The file is not ordered reliably by session or `try_seq`. Sequential profiling observed out-of-order and gapped groups, but exact per-session sequence checks must run after staging. `try_seq` need not be contiguous; uniqueness within a session is the integrity requirement unless authoritative upstream rules say otherwise.

## Payer-token evidence

Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. The CTO guide confirms that equality is meaningful only within a merchant: the same card receives a different token at another merchant. Under ADR 0005 they remain restricted internal values. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts, and they never establish a verified person.

## Import validation classification

### Fail the bootstrap before promotion

- Wrong/missing headers or unsupported schema version
- Unparseable required identifier, amount, or required timestamp
- Duplicate source `(session_key, try_seq)`
- A `try_seq = 0` row whose status is not `NoAttempt`, or a `NoAttempt` row whose sequence is not zero
- One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
- Terminal ownership/category conflicts
- Broken foreign-key construction or row-count reconciliation

### Preserve and report as quality evidence

- Documented nullable fields
- Timestamp-order anomalies
- Non-contiguous/out-of-order source rows or `try_seq` gaps
- Rare/unknown-but-well-formed source status/code values, pending the approved reference-code policy

### Never do

- Drop anomalous rows silently
- Replace missing values with plausible defaults
- Sum the repeated source fee at attempt grain
- Depend on CSV row order
- Insert a `NoAttempt` marker as a normalized payment attempt
- Claim payer tokens represent verified people