docs(import): define validation and recovery policy
This commit is contained in:
256
docs/IMPORT-VALIDATION-RECOVERY-SPEC.md
Normal file
256
docs/IMPORT-VALIDATION-RECOVERY-SPEC.md
Normal file
@@ -0,0 +1,256 @@
|
||||
# Import Validation, Rejection, and Recovery Specification
|
||||
|
||||
- Status: Decision-ready draft — stakeholder approval required
|
||||
- Capability: Data foundation bootstrap integrity
|
||||
- Depends on: Approved physical schema and one-time Docker import workflow
|
||||
- Implementation authorization: Not granted
|
||||
- Last updated: 2026-08-20
|
||||
|
||||
## Objective
|
||||
|
||||
Ensure that only the exact supplied challenge archive can become the immutable MVP PostgreSQL dataset, that normalization never silently loses or invents payment facts, and that interruption or malformed input cannot leave a partially trusted database. The workflow reports quality anomalies without exposing payer tokens or raw financial rows.
|
||||
|
||||
## Threat and failure model
|
||||
|
||||
### Trust boundaries
|
||||
|
||||
- The mounted gzip/CSV archive is untrusted input even though an operator placed it on the server.
|
||||
- CLI arguments and environment variables are untrusted configuration and cannot weaken validation.
|
||||
- Staging tables contain sensitive payment-related source data and are temporary security boundaries.
|
||||
- PostgreSQL normalized tables become trusted only after full validation, reconciliation, and atomic promotion.
|
||||
|
||||
### Assets protected
|
||||
|
||||
- Integrity and completeness of merchant analytics
|
||||
- Tenant relationships between categories, merchants, terminals, sessions, and attempts
|
||||
- Restricted opaque `payer_card_key` values
|
||||
- Database availability, disk capacity, and importer memory
|
||||
- Reproducible provenance and operator evidence
|
||||
|
||||
### Main abuse/failure cases
|
||||
|
||||
- Replacing the expected archive with a modified file
|
||||
- Gzip expansion, oversized records, malformed CSV, invalid UTF-8, or resource exhaustion
|
||||
- Duplicate or conflicting identities that corrupt normalization
|
||||
- Partial promotion after process/container/database failure
|
||||
- Sensitive values leaking through logs, reports, errors, or retained staging
|
||||
- Treating documented anomalies as invalid and silently losing valid source records
|
||||
|
||||
## Source identity and resource limits
|
||||
|
||||
The MVP pins this expected source identity:
|
||||
|
||||
```text
|
||||
filename: challenge_data.csv.gz
|
||||
schema: challenge-v1
|
||||
sha256: 84ac8a28df48ca7baeaf0b1cec563a3f0a3516039f5f62e9bfd11124ca35b461
|
||||
expected source rows: 2,213,289
|
||||
```
|
||||
|
||||
Preflight rejects a mismatch before creating staging data. The regular first-start interface cannot override the expected checksum, schema, or filename.
|
||||
|
||||
Initial fail-closed safety caps are:
|
||||
|
||||
| Limit | MVP value |
|
||||
|---|---:|
|
||||
| Compressed archive | 128 MiB |
|
||||
| Decompressed CSV bytes | 2 GiB |
|
||||
| Source data rows | 3,000,000 |
|
||||
| Decoded CSV record | 64 KiB |
|
||||
| Columns per record | exactly 22 |
|
||||
| Aggregate issue samples per rule | at most 20 source row numbers |
|
||||
|
||||
The supplied compressed archive is approximately 58.5 MiB, leaving operational headroom without accepting an unbounded gzip stream. Raising a limit requires a reviewed specification/config change; the import command cannot disable caps.
|
||||
|
||||
## Validation phases
|
||||
|
||||
Validation is ordered so cheap structural failures stop before expensive database work.
|
||||
|
||||
### Phase 1 — file preflight
|
||||
|
||||
- Require the fixed regular file mounted beneath `/import/input`; reject symlinks, directories, traversal, devices, pipes, and URLs.
|
||||
- Verify safe filename, compressed-size cap, gzip magic/integrity, decompression cap, exact lowercase SHA-256, and read-only access.
|
||||
- Decode UTF-8 strictly. A single UTF-8 BOM is permitted only before the first header name; replacement characters or invalid byte sequences are fatal.
|
||||
- Require the exact 22-column header names and order from `challenge-v1`; duplicate, missing, additional, or reordered headers are fatal.
|
||||
- Treat the first line as the only header. Blank data records and records with a non-exact field count are fatal.
|
||||
|
||||
### Phase 2 — row parsing and staging
|
||||
|
||||
Every row receives an internal `source_row_number` for diagnostics. The parser handles valid CSV quoting and embedded delimiters according to the chosen standard library; it never splits records manually.
|
||||
|
||||
Fatal row-level parse rules include:
|
||||
|
||||
- missing/blank required identifiers, category title, statuses, verification type, amount, fee, `created_at`, or `expire_in`;
|
||||
- identifier/control-text values containing NUL or disallowed control characters;
|
||||
- `try_seq` outside `0..32767`, amount outside `1..9223372036854775807`, negative fee, or negative duration;
|
||||
- invalid strict decimal integer syntax, timestamp syntax, or Tehran-local conversion;
|
||||
- `try_seq = 0` without `try_status = NoAttempt`, or `NoAttempt` with a nonzero sequence;
|
||||
- a no-attempt marker containing attempt-only PSP, attempt timestamp, bank/card, response, verification/settlement, or timing data inconsistent with the source contract;
|
||||
- a real attempt missing required `psp_code` or `try_created_at`;
|
||||
- only one of `issuer_bank_code` and `payer_card_key` populated;
|
||||
- verification type outside `Automated` or `Manual` for `challenge-v1`.
|
||||
|
||||
Nullable fields remain null. Empty strings in nullable source columns normalize to SQL null; whitespace is not silently trimmed from identifiers or codes. Required values may not become valid merely because trimming would fix them.
|
||||
|
||||
### Phase 3 — set-based relationship validation
|
||||
|
||||
After staging the whole archive, set-based PostgreSQL checks validate the dataset without relying on file order:
|
||||
|
||||
- unique `(session_key, try_seq)` source-record identity;
|
||||
- one category title per category id;
|
||||
- one category per merchant;
|
||||
- one merchant per terminal;
|
||||
- one verification method per terminal;
|
||||
- one merchant, terminal, category, amount, fee, session status, creation time, and expiry time per session;
|
||||
- terminal ownership agrees with the session merchant;
|
||||
- every real attempt resolves to exactly one staged session;
|
||||
- sequence values are unique within a session but need not be contiguous or sorted;
|
||||
- all foreign/reference entities can be constructed without ambiguity.
|
||||
|
||||
Any conflict is fatal for the entire import. The importer reports the violated rule and aggregate count, not conflicting raw values.
|
||||
|
||||
### Phase 4 — quality evidence that does not block promotion
|
||||
|
||||
The following are warnings and are preserved unchanged:
|
||||
|
||||
- `try_created_at` before session `created_at`;
|
||||
- `verified_at` before `try_created_at`;
|
||||
- `settled_at` before `verified_at`;
|
||||
- gaps or out-of-order attempt sequences;
|
||||
- documented nullable fields;
|
||||
- rare or unknown but syntactically well-formed lifecycle/response/reference codes, provided they do not violate a structural relationship.
|
||||
|
||||
Warnings never cause row deletion, timestamp correction, default substitution, or status reinterpretation. Unknown lifecycle statuses remain excluded from classified analytics until the metric catalogue explicitly classifies them.
|
||||
|
||||
### Phase 5 — exact reconciliation before promotion
|
||||
|
||||
For the pinned archive, all of these must match:
|
||||
|
||||
| Measure | Required value |
|
||||
|---|---:|
|
||||
| Source records | 2,213,289 |
|
||||
| No-attempt markers | 263,936 |
|
||||
| Real attempt records | 1,949,353 |
|
||||
| Distinct sessions | 2,062,839 |
|
||||
| Merchants | 343 |
|
||||
| Terminals | 348 |
|
||||
| Categories | 5 |
|
||||
|
||||
Additional equalities must hold:
|
||||
|
||||
```text
|
||||
source_records = no_attempt_markers + real_attempt_records
|
||||
normalized_payment_attempts = real_attempt_records
|
||||
normalized_payment_sessions = distinct_session_keys
|
||||
promoted session amount/fee count = normalized_payment_sessions
|
||||
```
|
||||
|
||||
The importer must also reconcile per-merchant session/attempt totals and per-status totals between staging and normalized projections. Any mismatch is fatal.
|
||||
|
||||
## Rejection policy
|
||||
|
||||
The MVP does not publish a partial dataset.
|
||||
|
||||
- One or more fatal issues cause the entire import run to fail.
|
||||
- Zero normalized payment facts from that run remain committed.
|
||||
- Valid-looking rows are not promoted around invalid rows.
|
||||
- `rejected_source_rows` records the count of rows directly violating fatal row rules; relationship/reconciliation failures are recorded separately and may invalidate more rows than can be safely attributed.
|
||||
- Warnings do not increment rejected rows and do not block promotion.
|
||||
- Public application services remain stopped after rejection.
|
||||
|
||||
This all-or-nothing policy is intentional because partial financial analytics could appear credible while producing incorrect merchant comparisons, rates, and totals.
|
||||
|
||||
## Promotion and atomicity
|
||||
|
||||
- Staging is run-scoped and cannot be queried by the API/worker database role.
|
||||
- Reference entities, merchants, terminals, sessions, attempts, calendar rows, demo profile hooks, and the final import-run state are promoted in one explicit PostgreSQL transaction where production-shaped testing confirms feasibility.
|
||||
- If atomic promotion must later be split for measured operational reasons, a replacement resumable protocol requires separate approval before implementation.
|
||||
- Constraint or transaction failure rolls back every normalized write from that promotion.
|
||||
- The `completed` state is written only after normalized counts and constraints reconcile inside the same commit boundary.
|
||||
- A completed import run is immutable.
|
||||
|
||||
## Issue reporting and redaction
|
||||
|
||||
`import_issues` and generated JSON/Markdown reports contain only:
|
||||
|
||||
- stable rule code and severity;
|
||||
- affected-row/conflict count;
|
||||
- at most 20 source row numbers;
|
||||
- affected column names;
|
||||
- safe numeric range/count summaries;
|
||||
- import-run id, checksum, schema/importer version, phase, and timestamps.
|
||||
|
||||
They never contain raw CSV records or source values for session, terminal, merchant, payer card, bank, PSP, response, amount, fee, username, password, or credentials. Routine logs use the same restriction. Operator errors name a rule and report location rather than echoing the rejected value.
|
||||
|
||||
## Interruption and staging cleanup
|
||||
|
||||
- Successful promotion drops/truncates its run-scoped staging immediately after the durable report is finalized.
|
||||
- A normal validation or promotion failure first persists safe aggregate issues, marks the run failed, then removes raw staging before exit.
|
||||
- Abrupt termination may leave an abandoned non-completed run and staging. On the next administrative importer invocation, the importer acquires a PostgreSQL advisory lock, marks stale non-terminal runs as interrupted, writes a safe issue, removes only staging linked to those run ids, and then exits with a recovery-required message.
|
||||
- No automatic cleanup touches completed normalized facts, another database/schema, or staging owned by an active locked importer.
|
||||
- Generated reports use atomic temporary-file rename; stale report temp files are safe to remove on the next invocation.
|
||||
|
||||
Because the original immutable archive remains available, raw failed staging has no long-term diagnostic purpose and is not retained.
|
||||
|
||||
## Concurrency contract
|
||||
|
||||
- A PostgreSQL advisory lock permits only one validate/import/recovery command for this dataset/database at a time.
|
||||
- A second invocation exits with `IMPORT_ALREADY_RUNNING`; it does not wait indefinitely or start another staging copy.
|
||||
- The lock is session-scoped so PostgreSQL releases it after process/container failure.
|
||||
- Import statements use the least-privileged dedicated importer role; runtime API/worker roles cannot create/drop staging or promote data.
|
||||
|
||||
## Recovery commands and behavior
|
||||
|
||||
The future importer exposes administrative commands through the same one-shot container:
|
||||
|
||||
```bash
|
||||
docker compose --profile bootstrap run --rm importer validate \
|
||||
--source /import/input/challenge_data.csv.gz \
|
||||
--schema-version challenge-v1
|
||||
|
||||
docker compose --profile bootstrap run --rm importer inspect \
|
||||
--run-id 42
|
||||
|
||||
docker compose --profile bootstrap run --rm importer recover \
|
||||
--run-id 42 \
|
||||
--source /import/input/challenge_data.csv.gz
|
||||
```
|
||||
|
||||
- `validate` performs file, row, relationship, quality, and reconciliation checks without promoting normalized facts.
|
||||
- `inspect` prints only safe run/issue/report metadata.
|
||||
- `recover` verifies that the named run is failed/interrupted, cleans its run-scoped staging, rechecks the pinned source identity, and starts a new import run from the beginning. It does not continue at a guessed CSV offset or reuse partially promoted facts.
|
||||
- Recovery of a completed run is rejected.
|
||||
- Dataset replacement, database truncation, and accepting a different checksum are not recovery operations and have no MVP command.
|
||||
|
||||
## Test strategy
|
||||
|
||||
- Table-driven unit tests for every fatal/warning rule and sensitive-value redaction.
|
||||
- Property/fuzz tests for CSV quoting, UTF-8, integer overflow, timestamps, gzip truncation/expansion, and record-size limits.
|
||||
- PostgreSQL integration fixtures for duplicate identities, cross-tenant relationship conflicts, exact null semantics, no-attempt normalization, and full rollback.
|
||||
- Fault injection at staging, validation, promotion, report-write, and completion-state boundaries.
|
||||
- Concurrency tests proving the advisory lock and stale-run behavior.
|
||||
- A full-archive rehearsal verifying pinned checksum, required counts, warning counts, peak memory/disk, cleanup, and a second-run no-op.
|
||||
|
||||
## Success criteria
|
||||
|
||||
- Only the pinned archive and exact schema can reach promotion.
|
||||
- No fatal import can leave normalized facts or start public services.
|
||||
- All source rows reconcile to normalized session/attempt grain without silent dropping or invention.
|
||||
- Known timestamp/sequence anomalies remain unchanged and appear as warnings.
|
||||
- Reports and logs contain no restricted source values or raw rows.
|
||||
- Success, expected failure, process kill, database loss, disk exhaustion, and concurrent invocation all end in a deterministic recoverable state.
|
||||
- Recovery starts from the immutable archive and cannot modify completed facts.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Always: pin source identity, enforce resource caps, validate in phases, fail the complete import on fatal issues, preserve warnings, use an advisory lock, redact evidence, and remove temporary raw staging.
|
||||
- Ask first: change a fatal/warning classification, expected count/checksum, size cap, transaction boundary, issue-retention field, or recovery behavior.
|
||||
- Never: partially publish, skip checks, accept another checksum, retain raw failed rows, log source values, guess missing data, mutate completed facts, or recover by truncating normalized tables.
|
||||
|
||||
## Decision batch for approval
|
||||
|
||||
1. Pin the exact supplied archive SHA-256, schema, filename, and expected reconciliation counts.
|
||||
2. Fail the whole import on any fatal issue; never publish a partial dataset.
|
||||
3. Preserve documented timestamp/sequence/null anomalies as warnings without changing rows.
|
||||
4. Remove raw staging after success/failure and clean abandoned run-scoped staging on the next administrative invocation, retaining only redacted aggregate evidence.
|
||||
5. Enforce the proposed resource caps and make recovery restart from the immutable archive rather than resume partial CSV/database state.
|
||||
Reference in New Issue
Block a user