From 9d57be161d0ca737aeb0eed0db24c6ddce1afc3c Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 13:38:23 +0330 Subject: [PATCH] docs: define calendar-aware analytics model --- AGENTS.md | 1 + CONTEXT.md | 54 +++++++++++++++++++ docs/DATA-FOUNDATION-SPEC.md | 42 ++++++++++++++- ...07-calendar-aware-analytics-projections.md | 32 +++++++++++ 4 files changed, 127 insertions(+), 2 deletions(-) create mode 100644 CONTEXT.md create mode 100644 docs/decisions/0007-calendar-aware-analytics-projections.md diff --git a/AGENTS.md b/AGENTS.md index 90d00b4..b7d5263 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,6 +8,7 @@ For every Go-related task, always invoke and follow the `golang-how-to` skill, t ## Product invariants +- Use the canonical domain terms in `CONTEXT.md`; update that glossary when a term's meaning is approved, without adding implementation details to it. - The backend is the authority for authentication, authorization, tenant isolation, metric definitions, and data provenance. - Every merchant-scoped query derives merchant identity from the authenticated server-side session; never trust a client-supplied merchant identifier as authorization. - Cross-merchant results are anonymized aggregates only. Exclude the active merchant and suppress the benchmark unless at least 10 other merchants and 1,000 combined peer sessions contribute to the approved cohort/window. diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..e3aba5f --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,54 @@ +# Domain Glossary + +## Merchant + +A business represented in the source data. A merchant owns terminals, payment sessions, and merchant users. In the MVP, detailed data is visible only within that merchant's authenticated boundary. + +## Merchant User + +A person/account permitted to authenticate as one merchant. The hackathon seed creates an initial demo user for each merchant; a merchant and a merchant user are not the same concept. + +## Terminal + +A merchant-owned payment endpoint identified by the source terminal key. + +## Payment Session + +One payment journey identified by the source session key. A session belongs to one merchant and terminal and can contain one or more payment attempts. + +## Payment Attempt + +One try within a payment session, distinguished by its try sequence. This is the grain of a source CSV row and must not be confused with a unique payment session. + +## Opaque Payer Token + +The nullable `payer_card_key` source value. It is an unverified, merchant-scoped token used only for internal approximate repeat-payer analytics. It is not a card number or verified person identifier. + +## Import Run + +One traceable attempt to ingest a specific source archive/schema using a specific importer revision. It records outcome, counts, validation evidence, and provenance. + +## Calendar Date + +One civil date enriched with Gregorian and Jalali calendar attributes, Persian month/season, weekday, and reporting attributes. + +## Calendar Event + +A curated, versioned named period relevant to analysis, such as Nowruz or another Iranian holiday. Event membership is evidence supplied by the calendar model, not inferred by an LLM. + +## Reporting Period + +An approved time window used for metrics: day, Jalali month, Persian season, calendar event, or another explicitly defined comparison window. + +## Analytical Metric + +A deterministic numerical measure with a definition, source fields, filters, period, unit, missing-data behavior, and drill-down path. + +## Analytics Projection + +A rebuildable summary derived from normalized source-of-truth records to answer an approved metric efficiently. A projection is not independent business truth. + +## Peer Cohort + +An approved group of other merchants used for anonymized comparison. The active merchant is excluded, and a benchmark is suppressed unless at least 10 peers and 1,000 peer sessions contribute. + diff --git a/docs/DATA-FOUNDATION-SPEC.md b/docs/DATA-FOUNDATION-SPEC.md index 45b016c..fe12b21 100644 --- a/docs/DATA-FOUNDATION-SPEC.md +++ b/docs/DATA-FOUNDATION-SPEC.md @@ -13,6 +13,8 @@ Repeatably ingest the supplied payment-attempt archive into PostgreSQL, preserve - Streamed `.csv.gz` ingestion from a local, Git-ignored input path - Import runs, checksums, schema version, row counts, rejection/error reporting - Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts +- Versioned Gregorian/Jalali calendar dates and curated Iranian event/holiday periods +- Rebuildable daily base summaries and metric-specific period projections for approved analytics - Synthetic merchant display profiles and one demo account per merchant - Idempotent re-runs and explicit import lifecycle - Database constraints/indexes driven by documented access paths @@ -36,6 +38,10 @@ erDiagram MERCHANT ||--o{ PAYMENT_SESSION : receives TERMINAL ||--o{ PAYMENT_SESSION : handles PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains + CALENDAR_DATE ||--o{ CALENDAR_DATE_EVENT : includes + CALENDAR_EVENT ||--o{ CALENDAR_DATE_EVENT : occurs_on + CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups + MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by IMPORT_RUN ||--o{ IMPORT_ISSUE : reports @@ -111,6 +117,34 @@ erDiagram bigint affected_rows jsonb evidence } + CALENDAR_DATE { + date date PK + int jalali_year + int jalali_month + int jalali_day + text persian_season + int iso_weekday + } + CALENDAR_EVENT { + bigint id PK + text code UK + text title_fa + text event_type + text source_version + } + CALENDAR_DATE_EVENT { + date date FK + bigint calendar_event_id FK + } + MERCHANT_DAY_SUMMARY { + bigint merchant_id FK + date local_date FK + bigint session_count + bigint attempt_count + bigint successful_count + bigint amount_sum + bigint source_import_run_id FK + } ``` This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved. @@ -124,6 +158,7 @@ This ERD is a logical baseline, not an approved physical schema. In particular, 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. +8. Populate and validate the approved calendar range, then rebuild affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh. ## Synthetic merchant/account rules @@ -143,6 +178,9 @@ This ERD is a logical baseline, not an approved physical schema. In particular, - Missing values remain missing rather than being silently replaced with factual-looking defaults. - The import report captures checksum, importer revision, timings, counts, validation failures, and quality metrics. - Raw data and database state remain ignored by Git; code, schema, rules, docs, and safe templates are tracked. +- Daily, Jalali-month, Persian-season, and approved calendar-event analytics reconcile to deterministic source queries for additive metrics. +- Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. +- Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. ## Approval gates / unresolved decisions @@ -158,6 +196,6 @@ Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opa Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006. -Remaining approval gates: +Resolved: use normalized source-of-truth entities plus versioned calendar/event dimensions, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Monthly, Persian-season, and holiday results use explicit calendar definitions. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. -1. Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone. +No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, calendar source, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec. diff --git a/docs/decisions/0007-calendar-aware-analytics-projections.md b/docs/decisions/0007-calendar-aware-analytics-projections.md new file mode 100644 index 0000000..dd3ae8e --- /dev/null +++ b/docs/decisions/0007-calendar-aware-analytics-projections.md @@ -0,0 +1,32 @@ +# ADR 0007: Calendar-aware analytics projections without initial table partitioning + +- Status: Accepted +- Date: 2026-08-20 + +## Context + +The source contains approximately 2.2 million payment-attempt rows, but attempts are only one domain entity. Merchant, terminal, session, attempt, import, identity, and calendar concepts require separate normalized relations connected by constraints. The product also needs daily, Jalali-month, Persian-season, and Iranian holiday/event analysis, including evidence that may later support bounded AI recommendations. + +Calculating every dashboard result from normalized attempt rows on every request creates avoidable latency. Conversely, splitting attempts into unrelated tables by month/category or adding PostgreSQL partitions before measuring query and maintenance behavior introduces complexity without evidence. Some metrics are additive across days, while distinct payer-token counts and percentiles are not. + +## Decision + +Keep normalized business/source-of-truth relations: categories, merchants, merchant users, terminals, payment sessions, payment attempts, and import provenance. The attempt relation retains one row per source attempt; normalization does not arbitrarily distribute a single entity's rows across category/month tables. + +Add a versioned calendar model containing Gregorian and Jalali date attributes, Persian seasons, weekdays, and curated named events such as Nowruz. Calendar-event definitions have explicit provenance/version and are deterministic inputs to analytics, not LLM-generated labels. + +Build idempotent, rebuildable daily base projections containing additive components such as session count, attempt count, successful count, amount sum, and fee sum as approved by the metric catalogue. Derive month/season/event totals by grouping these components through the calendar model and recompute ratios from their numerators and denominators. + +Create separate metric-specific period projections only for non-additive metrics or query paths proven expensive—for example, exact unique payer-token counts cannot be obtained by summing daily distinct counts. Projection rows record source import/version and freshness. + +Keep normalized fact tables unpartitioned initially. Add targeted indexes from approved API access paths and verified query plans. Introduce PostgreSQL partitioning only through a future ADR supported by measured table growth, query latency, vacuum/maintenance behavior, or retention operations. + +## Consequences + +- The normalized ERD remains semantically correct while dashboards read smaller, purpose-built projections. +- Daily components support many calendar groupings without precomputing every possible period. +- Calendar definitions and metric provenance make future AI observations traceable. +- A single observed Nowruz period may be reported as an observation but not claimed as recurring seasonality; multiple comparable years are needed for that claim. +- Projection refresh, failure, stale-state visibility, and reconciliation tests become required operational behavior. +- Schema/index/projection details remain subject to data profiling and approved metric/API requirements. +