Challenge Dataset Profiling Report

Status: Evidence baseline for specification
Profiled: 2026-08-20
Source SHA-256: 84AC8A28DF48CA7BAEAF0B1CEC563A3F0A3516039F5F62E9BFD11124CA35B461
Source archive: Git-ignored data/incoming/challenge_data.csv.gz

Method

The compressed CSV was scanned without modifying it. Counts and value distributions are descriptive evidence, not source instructions. Relationship/uniqueness constraints must be revalidated by the importer against staging before promotion. A quick sequential session scan observed 2,069,061 session groups versus 2,062,839 distinct session keys, proving that rows for a session are not guaranteed to be contiguous; implementation must never depend on file order.

Shape and grain

Measure	Observed
CSV source records	2,213,289
`NoAttempt` session markers (not normalized attempts)	263,936
Actual attempt records (`try_seq >= 1`)	1,949,353
Distinct payment sessions	2,062,839
Merchants	343
Terminals	348
Categories	5
Distinct populated payer tokens	402,173
Source period	2026-01-01 00:00:08 through 2026-06-30 23:59:55

The source record grain is an attempt-or-no-attempt session record. (session_key, try_seq) is the candidate source-record identity. The CTO guide defines (try_seq = 0, try_status = NoAttempt) as a marker for a session with no recorded attempt; those 263,936 rows create sessions but not normalized attempts. Sessions and actual attempts must remain different domain concepts.

Nullability

Column	Missing rows	Approx. missing
`switch_response_code`	2,155,696	97.4%
`issuer_bank_code`	1,178,921	53.3%
`payer_card_key`	1,178,921	53.3%
`verify_time_ms`	1,179,013	53.3%
`verified_at`	1,125,164	50.8%
`settled_at`	1,116,034	50.4%
`psp_code`	263,936	11.9%
`try_created_at`	263,936	11.9%
`init_time_ms`	346,747	15.7%

Missing optional values are analytically meaningful and must remain null. Dashboards and metric definitions must disclose coverage where it affects interpretation.

Controlled vocabularies

Session status: Failed, Verified, Paid, Reversed (one observed reversed row).
Attempt status: NoAttempt, InBank, Failed, Verified, Paid, Reversed.
Verification type: Automated, Manual.
PSP codes: 8 observed opaque codes.
Issuer-bank codes: 31 observed opaque codes.
Switch response codes: 84 observed values and extremely sparse.

These are source codes, not Persian user-facing descriptions. Meanings and success semantics belong in a versioned metric/reference catalogue; they must not be guessed from labels alone.

Numeric ranges and types

Source field	Observed range	Proposed canonical type/meaning
`try_seq`	0–135	source `smallint`; `0` is a no-attempt marker, normalized attempts require `1–135`
`amount`	1,000–2,000,000,000	`bigint`, Rial
`adjusted_fee`	1,920–284,800	positive integer challenge stand-in, uniformly confidentiality-scaled; mapped to the product's canonical real-fee field `fee_amount_rial` per ADR 0013
`init_time_ms`	44–59,905	nullable integer milliseconds
`verify_time_ms`	35–58,553	nullable integer milliseconds
`expire_in`	timestamp strings	rename canonically to `expires_at`; it is not a duration

No negative or zero amount/fee/timing values were observed. Source identifiers are stored as text even when they look numeric so leading zeros and upstream formatting remain intact.

Relationship evidence

Each observed category id has one Persian title.
The quick scan found no merchant-to-category or terminal-to-owner conflict, but exact staging validation remains mandatory because session rows are not contiguous.
Merchant volume is highly skewed: median merchant has 78 attempts, p95 has 8,915, and the largest has 1,138,592. The MVP therefore uses a merchant-weighted median—one metric value per eligible peer merchant—to represent a typical same-category peer. Transaction-weighted peer aggregates are excluded because the largest merchants would dominate them.
One terminal/merchant/category, amount/session, and fee/session relationship are proposed invariants, subject to exact staging validation before constraints are finalized.
All 150,450 rows repeating a previously observed session repeated the same adjusted_fee; zero fee conflicts were found. The normalized model therefore stores the fee once per session rather than once per attempt.
The CTO guide defines psp_code at attempt grain and namespaces switch_response_code by PSP; identical raw response codes from different PSPs are not semantically interchangeable.
A complete 2,213,289-row scan found verify_type invariant within every terminal (zero of 348 terminals had multiple values). One merchant had more than one value because it owned terminals configured differently. The normalized physical model therefore stores verification type on terminals, not payment_attempts or merchants.
Maximum observed source text lengths were measured for type/index review: session key 7, terminal key 4, merchant key 4, category key 8, category title 34, session status 8, attempt status 9, switch response 11, PSP code 6, bank code 7, payer token 11, and verification type 9. Physical source identifiers remain text so opaque upstream formats are not frozen to challenge-only maxima.

Timestamp quality

Source timestamps are timezone-naive and are interpreted as Asia/Tehran under ADR 0004. Lexical ordering is valid for the fixed source format, and profiling observed:

234,653 rows where try_created_at precedes session created_at.
787,573 rows where settled_at precedes verified_at.
6 rows where verified_at precedes try_created_at.

Therefore timestamp ordering is a quality signal, not a hard rejection constraint. The importer preserves values, reports anomaly counts, and must not silently reorder or “fix” them.

Attempt sequence quality

The file is not ordered reliably by session or try_seq. Sequential profiling observed out-of-order and gapped groups, but exact per-session sequence checks must run after staging. try_seq need not be contiguous; uniqueness within a session is the integrity requirement unless authoritative upstream rules say otherwise.

Payer-token evidence

Populated payer_card_key values are opaque alphanumeric tokens 6–11 characters long. The CTO guide confirms that equality is meaningful only within a merchant: the same card receives a different token at another merchant. Under ADR 0005 they remain restricted internal values. Their 53.3% missing rate prevents presenting distinct-token counts as complete customer counts, and they never establish a verified person.

Import validation classification

The implementation-ready classification, pinned source identity, resource caps, redacted evidence, and recovery behavior are specified in IMPORT-VALIDATION-RECOVERY-SPEC.md. This section remains the evidence summary behind that contract.

Fail the bootstrap before promotion

Wrong/missing headers or unsupported schema version
Unparseable required identifier, amount, or required timestamp
Duplicate source (session_key, try_seq)
A try_seq = 0 row whose status is not NoAttempt, or a NoAttempt row whose sequence is not zero
One source session mapping to multiple merchants/terminals/categories/amounts after exact staging validation
Terminal ownership/category conflicts
Broken foreign-key construction or row-count reconciliation

Preserve and report as quality evidence

Documented nullable fields
Timestamp-order anomalies
Non-contiguous/out-of-order source rows or try_seq gaps
Rare/unknown-but-well-formed source status/code values, pending the approved reference-code policy

Never do

Drop anomalous rows silently
Replace missing values with plausible defaults
Sum the repeated source fee at attempt grain
Depend on CSV row order
Insert a NoAttempt marker as a normalized payment attempt
Claim payer tokens represent verified people

8.1 KiB Raw Permalink Blame History Unescape Escape