From f472ad5b1d244eda63ded390ded64bf0f2e4ae2a Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 13:59:36 +0330 Subject: [PATCH] docs: record production analytics decisions --- docs/DATA-FOUNDATION-SPEC.md | 17 +++++++++ docs/METRIC-CATALOGUE.md | 37 +++++++++++++++++++ docs/PHYSICAL-SCHEMA-PROPOSAL.md | 7 +++- ...roduction-write-and-analytics-freshness.md | 25 +++++++++++++ .../0009-versioned-iranian-calendar.md | 24 ++++++++++++ 5 files changed, 108 insertions(+), 2 deletions(-) create mode 100644 docs/METRIC-CATALOGUE.md create mode 100644 docs/decisions/0008-production-write-and-analytics-freshness.md create mode 100644 docs/decisions/0009-versioned-iranian-calendar.md diff --git a/docs/DATA-FOUNDATION-SPEC.md b/docs/DATA-FOUNDATION-SPEC.md index 41cedb0..a325613 100644 --- a/docs/DATA-FOUNDATION-SPEC.md +++ b/docs/DATA-FOUNDATION-SPEC.md @@ -165,6 +165,15 @@ This contract applies only to the one-time challenge-data bootstrap and explicit 8. Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh. 9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract. +## Production write and analytics freshness contract + +- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema. +- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow. +- Live rows are distinguishable from bootstrap rows and include stable creation/update metadata for incremental analytics. Bootstrap-only import provenance is not forged for live rows. +- Incremental analytics projections refresh within 15 minutes of an eligible source change. +- A nightly idempotent reconciliation checks/recomputes the approved authoritative window and detects drift. +- Failed refreshes never advance the API/UI freshness watermark. + ## Synthetic merchant/account rules - Create exactly one initial demo user per distinct merchant unless the later identity spec explicitly changes cardinality. @@ -186,6 +195,8 @@ This contract applies only to the one-time challenge-data bootstrap and explicit - Daily, Jalali-month, Persian-season, and approved calendar-event analytics reconcile to deterministic source queries for additive metrics. - Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. - Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. +- Incremental projections become queryable within 15 minutes of eligible live writes, and nightly reconciliation detects/corrects drift without double counting. +- Calendar rows covering the reporting period reconcile with reviewed official snapshots and record provenance/checksums. ## Approval gates / unresolved decisions @@ -203,4 +214,10 @@ Resolved: peer benchmarks exclude the active merchant and require at least 10 ot Resolved: use normalized source-of-truth entities plus versioned calendar/event dimensions, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Monthly, Persian-season, and holiday results use explicit calendar definitions. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. +Resolved: ZarinPal writes future production rows directly to the normalized tables. Our backend owns schema/read/analytics contracts, refreshes projections within 15 minutes, and performs nightly reconciliation. CSV ingestion remains one-time only. See ADR 0008. + +Resolved: commit reviewed yearly Iranian calendar snapshots sourced from the University of Tehran Institute of Geophysics Calendar Center; do not call a calendar API at runtime. Official holidays and analytical campaign/event windows remain distinguishable and versioned. See ADR 0009. + +Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`. + No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, calendar source, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec. diff --git a/docs/METRIC-CATALOGUE.md b/docs/METRIC-CATALOGUE.md new file mode 100644 index 0000000..ab56586 --- /dev/null +++ b/docs/METRIC-CATALOGUE.md @@ -0,0 +1,37 @@ +# Analytics Metric Catalogue + +- Status: Draft — definitions are approved individually +- Version: 0.1.0 +- Last updated: 2026-08-20 + +Every merchant-facing number must identify its grain, formula, source fields, filters, time basis, unit, missing-data behavior, trace path, and definition version. + +## Confirmed successful sessions + +- Canonical id: `confirmed_successful_sessions` +- Grain: distinct payment session +- Formula: count payment sessions where source `session_status = 'Verified'` +- Unit: sessions +- Time attribution: session `created_at`, converted to `Asia/Tehran` local reporting date +- Missing behavior: `session_status` is required; unknown values are not counted and trigger quality evidence +- Traceability: drill down to eligible session identifiers and their attempt histories within the authenticated merchant +- Approved decision: `Paid` is not included; it remains a separate state until ZarinPal provides an authoritative definition + +## Paid but not confirmed as verified + +- Canonical id: `paid_unverified_sessions` +- Grain: distinct payment session +- Formula: count payment sessions where source `session_status = 'Paid'` +- Unit: sessions +- Presentation: separate operational state; never label as confirmed success or failure +- Traceability: same merchant-scoped session/attempt drill-down + +## Open definitions + +- Success-rate denominator and treatment of `Failed`/`Reversed`/`Paid` +- Amount metric inclusion by session status +- Retry and recovery definitions +- Timing percentile definitions and missing coverage +- Fee-ratio terminology for obfuscated `adjusted_fee` +- Merchant-weighted versus transaction-weighted peer comparisons + diff --git a/docs/PHYSICAL-SCHEMA-PROPOSAL.md b/docs/PHYSICAL-SCHEMA-PROPOSAL.md index 54b7749..ca6587c 100644 --- a/docs/PHYSICAL-SCHEMA-PROPOSAL.md +++ b/docs/PHYSICAL-SCHEMA-PROPOSAL.md @@ -51,6 +51,8 @@ Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Creden `adjusted_fee` uses fixed precision and is labeled as an obfuscated relative value. The physical precision is finalized from exact decimal profiling before migration approval. +Both fact tables distinguish `bootstrap` from `zarinpal_live` origin. Bootstrap rows reference an import run; live rows do not invent one. Live creation/update metadata forms the incremental analytics watermark contract, including status changes to an existing session. + ### Rebuildable analytics projections | Table family | Grain | Purpose | @@ -62,6 +64,8 @@ Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Creden Projection names/columns are finalized by the analytics metric catalogue. Avoid a generic key/value metrics table because it weakens types, constraints, and discoverability. +Projection metadata records source watermark, calculation version, calculated time, and status. An incremental refresh runs at least every 15 minutes; a nightly reconciliation checks/rebuilds an authoritative window and advances freshness only after successful atomic publication. + ## Initial index policy - Unique indexes implement source identities and natural relationship constraints. @@ -80,5 +84,4 @@ Projection names/columns are finalized by the analytics metric catalogue. Avoid ## Production population boundary -After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records to PostgreSQL. The exact contract is still open: ZarinPal may write this normalized schema directly or expose an existing authoritative schema for read-only analytics. This project will not invent recurring ETL until that boundary is answered. - +After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records directly to the normalized PostgreSQL session/attempt tables. This project owns the schema/migration and analytics contract but not ZarinPal's write workflow. Compatibility, required fields, source origin, and creation/update watermarks are versioned interfaces; recurring CSV ETL is prohibited. diff --git a/docs/decisions/0008-production-write-and-analytics-freshness.md b/docs/decisions/0008-production-write-and-analytics-freshness.md new file mode 100644 index 0000000..3f0d7c3 --- /dev/null +++ b/docs/decisions/0008-production-write-and-analytics-freshness.md @@ -0,0 +1,25 @@ +# ADR 0008: ZarinPal owns live writes; analytics refresh within 15 minutes + +- Status: Accepted +- Date: 2026-08-20 + +## Context + +The challenge archive is a one-time bootstrap source. In production, ZarinPal already creates payment data and will write future records directly to the normalized PostgreSQL schema. Building a recurring CSV importer, polling source, or competing payment-write workflow would duplicate upstream responsibility. Derived dashboard projections still need bounded freshness and correction when live source rows arrive or change status. + +## Decision + +ZarinPal is the authoritative producer of live `payment_sessions` and `payment_attempts`. This backend owns the normalized schema contract, versioned migrations, analytical reads/projections, and tenant-safe APIs, but not the business workflow that produces live payment records. + +Fact rows identify `bootstrap` versus `zarinpal_live` origin. Bootstrap rows reference import provenance; live rows do not fabricate import runs. The live-write contract supplies stable creation/update metadata so an idempotent high-watermark process can include new rows and status updates. + +Refresh affected analytics projections within 15 minutes. Run a nightly idempotent reconciliation over an approved authoritative window to detect and correct missed/changed rows. Publish projections atomically and advance the visible freshness watermark only after success. + +## Consequences + +- No recurring CSV ingestion service, file watcher, or source polling is built. +- Schema migrations require compatibility coordination with ZarinPal's writer. +- Analytics refresh is separate from ingestion and requires scheduling, checkpoints, idempotency, metrics, and stale-state UX. +- If upstream cannot provide reliable update metadata, revise the refresh algorithm/window before implementation rather than silently missing mutations. +- The static hackathon dataset builds projections once after bootstrap; production scheduling remains testable with synthetic live writes. + diff --git a/docs/decisions/0009-versioned-iranian-calendar.md b/docs/decisions/0009-versioned-iranian-calendar.md new file mode 100644 index 0000000..8e2c141 --- /dev/null +++ b/docs/decisions/0009-versioned-iranian-calendar.md @@ -0,0 +1,24 @@ +# ADR 0009: Version reviewed Iranian calendar snapshots in Git + +- Status: Accepted +- Date: 2026-08-20 + +## Context + +Monthly, Persian-season, Nowruz, and public-holiday analytics need deterministic Gregorian/Jalali mappings and named date ranges. Runtime dependence on a public calendar API would add availability, version drift, and reproducibility risks. Official holiday dates can change across years, especially events derived from other calendars. + +## Decision + +Commit a reviewed, machine-readable calendar snapshot for every supported Jalali year. Use the Calendar Center of the University of Tehran Institute of Geophysics as the authoritative annual source (`https://calendar.ut.ac.ir/`). The source URL/document identity, retrieval date, checksum, reviewer, and snapshot schema version accompany each file. + +For the challenge period, cover both Jalali 1404 and 1405 because Gregorian January–June 2026 crosses the Jalali year boundary. Runtime code reads validated database calendar tables populated from these versioned snapshots and never calls an external calendar API. + +Differentiate official public holidays from analytical event/campaign windows. A Nowruz analytical window may extend beyond officially closed days, but its exact dates, rationale, and version are explicit rather than inferred by an LLM. + +## Consequences + +- Historical analytics are reproducible even if websites/APIs change. +- Annual calendar updates are reviewed source changes with validation tests. +- Unsupported years fail clearly instead of guessing holiday membership. +- Calendar data is committed reference data; raw payment archives remain ignored. +