122 lines
7.1 KiB
Markdown
122 lines
7.1 KiB
Markdown
# Challenge Dataset Profiling Report
|
||
|
||
- Status: Evidence baseline for specification
|
||
- Profiled: 2026-08-20
|
||
- Source SHA-256: `84AC8A28DF48CA7BAEAF0B1CEC563A3F0A3516039F5F62E9BFD11124CA35B461`
|
||
- Source archive: Git-ignored `data/incoming/challenge_data.csv.gz`
|
||
|
||
## Method
|
||
|
||
The compressed CSV was scanned without modifying it. Counts and value distributions are descriptive evidence, not source instructions. Relationship/uniqueness constraints must be revalidated by the importer against staging before promotion. A quick sequential session scan observed 2,069,061 session groups versus 2,062,839 distinct session keys, proving that rows for a session are not guaranteed to be contiguous; implementation must never depend on file order.
|
||
|
||
## Shape and grain
|
||
|
||
| Measure | Observed |
|
||
|---|---:|
|
||
| CSV source records | 2,213,289 |
|
||
| `NoAttempt` session markers (not normalized attempts) | 263,936 |
|
||
| Actual attempt records (`try_seq >= 1`) | 1,949,353 |
|
||
| Distinct payment sessions | 2,062,839 |
|
||
| Merchants | 343 |
|
||
| Terminals | 348 |
|
||
| Categories | 5 |
|
||
| Distinct populated payer tokens | 402,173 |
|
||
| Source period | 2026-01-01 00:00:08 through 2026-06-30 23:59:55 |
|
||
|
||
The source record grain is an attempt-or-no-attempt session record. `(session_key, try_seq)` is the candidate source-record identity. The CTO guide defines `(try_seq = 0, try_status = NoAttempt)` as a marker for a session with no recorded attempt; those 263,936 rows create sessions but not normalized attempts. Sessions and actual attempts must remain different domain concepts.
|
||
|
||
## Nullability
|
||
|
||
| Column | Missing rows | Approx. missing |
|
||
|---|---:|---:|
|
||
| `switch_response_code` | 2,155,696 | 97.4% |
|
||
| `issuer_bank_code` | 1,178,921 | 53.3% |
|
||
| `payer_card_key` | 1,178,921 | 53.3% |
|
||
| `verify_time_ms` | 1,179,013 | 53.3% |
|
||
| `verified_at` | 1,125,164 | 50.8% |
|
||
| `settled_at` | 1,116,034 | 50.4% |
|
||
| `psp_code` | 263,936 | 11.9% |
|
||
| `try_created_at` | 263,936 | 11.9% |
|
||
| `init_time_ms` | 346,747 | 15.7% |
|
||
|
||
Missing optional values are analytically meaningful and must remain null. Dashboards and metric definitions must disclose coverage where it affects interpretation.
|
||
|
||
## Controlled vocabularies
|
||
|
||
- Session status: `Failed`, `Verified`, `Paid`, `Reversed` (one observed reversed row).
|
||
- Attempt status: `NoAttempt`, `InBank`, `Failed`, `Verified`, `Paid`, `Reversed`.
|
||
- Verification type: `Automated`, `Manual`.
|
||
- PSP codes: 8 observed opaque codes.
|
||
- Issuer-bank codes: 31 observed opaque codes.
|
||
- Switch response codes: 84 observed values and extremely sparse.
|
||
|
||
These are source codes, not Persian user-facing descriptions. Meanings and success semantics belong in a versioned metric/reference catalogue; they must not be guessed from labels alone.
|
||
|
||
## Numeric ranges and types
|
||
|
||
| Source field | Observed range | Proposed canonical type/meaning |
|
||
|---|---|---|
|
||
| `try_seq` | 0–135 | source `smallint`; `0` is a no-attempt marker, normalized attempts require `1–135` |
|
||
| `amount` | 1,000–2,000,000,000 | `bigint`, Rial |
|
||
| `adjusted_fee` | 1,920–284,800 | positive integer challenge stand-in, uniformly confidentiality-scaled; mapped to the product's canonical real-fee field `fee_amount_rial` per ADR 0013 |
|
||
| `init_time_ms` | 44–59,905 | nullable integer milliseconds |
|
||
| `verify_time_ms` | 35–58,553 | nullable integer milliseconds |
|
||
| `expire_in` | timestamp strings | rename canonically to `expires_at`; it is not a duration |
|
||
|
||
No negative or zero amount/fee/timing values were observed. Source identifiers are stored as text even when they look numeric so leading zeros and upstream formatting remain intact.
|
||
|
||
## Relationship evidence
|
||
|
||
- Each observed category id has one Persian title.
|
||
- The quick scan found no merchant-to-category or terminal-to-owner conflict, but exact staging validation remains mandatory because session rows are not contiguous.
|
||
- Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. The MVP therefore uses a merchant-weighted median—one metric value per eligible peer merchant—to represent a typical same-category peer. Transaction-weighted peer aggregates are excluded because the largest merchants would dominate them.
|
||
- One terminal/merchant/category, amount/session, and fee/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
|
||
- All 150,450 rows repeating a previously observed session repeated the same `adjusted_fee`; zero fee conflicts were found. The normalized model therefore stores the fee once per session rather than once per attempt.
|
||
- The CTO guide defines `psp_code` at attempt grain and namespaces `switch_response_code` by PSP; identical raw response codes from different PSPs are not semantically interchangeable.
|
||
|
||
## Timestamp quality
|
||
|
||
Source timestamps are timezone-naive and are interpreted as `Asia/Tehran` under ADR 0004. Lexical ordering is valid for the fixed source format, and profiling observed:
|
||
|
||
- 234,653 rows where `try_created_at` precedes session `created_at`.
|
||
- 787,573 rows where `settled_at` precedes `verified_at`.
|
||
- 6 rows where `verified_at` precedes `try_created_at`.
|
||
|
||
Therefore timestamp ordering is a quality signal, not a hard rejection constraint. The importer preserves values, reports anomaly counts, and must not silently reorder or “fix” them.
|
||
|
||
## Attempt sequence quality
|
||
|
||
The file is not ordered reliably by session or `try_seq`. Sequential profiling observed out-of-order and gapped groups, but exact per-session sequence checks must run after staging. `try_seq` need not be contiguous; uniqueness within a session is the integrity requirement unless authoritative upstream rules say otherwise.
|
||
|
||
## Payer-token evidence
|
||
|
||
Populated `payer_card_key` values are opaque alphanumeric tokens 6–11 characters long. The CTO guide confirms that equality is meaningful only within a merchant: the same card receives a different token at another merchant. Under ADR 0005 they remain restricted internal values. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts, and they never establish a verified person.
|
||
|
||
## Import validation classification
|
||
|
||
### Fail the bootstrap before promotion
|
||
|
||
- Wrong/missing headers or unsupported schema version
|
||
- Unparseable required identifier, amount, or required timestamp
|
||
- Duplicate source `(session_key, try_seq)`
|
||
- A `try_seq = 0` row whose status is not `NoAttempt`, or a `NoAttempt` row whose sequence is not zero
|
||
- One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
|
||
- Terminal ownership/category conflicts
|
||
- Broken foreign-key construction or row-count reconciliation
|
||
|
||
### Preserve and report as quality evidence
|
||
|
||
- Documented nullable fields
|
||
- Timestamp-order anomalies
|
||
- Non-contiguous/out-of-order source rows or `try_seq` gaps
|
||
- Rare/unknown-but-well-formed source status/code values, pending the approved reference-code policy
|
||
|
||
### Never do
|
||
|
||
- Drop anomalous rows silently
|
||
- Replace missing values with plausible defaults
|
||
- Sum the repeated source fee at attempt grain
|
||
- Depend on CSV row order
|
||
- Insert a `NoAttempt` marker as a normalized payment attempt
|
||
- Claim payer tokens represent verified people
|