From 680596fe73be4249fe73dbd98e8b98d42b4871cb Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 16:17:27 +0330 Subject: [PATCH] docs: adopt lazy analytics caching --- CONTEXT.md | 4 +++ README.md | 2 +- docs/DATA-FOUNDATION-SPEC.md | 29 ++++++++++++------- docs/PHYSICAL-SCHEMA-PROPOSAL.md | 11 ++++--- docs/decisions/0003-ent-pgx-data-access.md | 6 ++-- ...07-calendar-aware-analytics-projections.md | 2 +- ...roduction-write-and-analytics-freshness.md | 25 ---------------- .../decisions/0010-limit-reporting-periods.md | 2 +- .../0011-lazy-merchant-analytics-cache.md | 26 +++++++++++++++++ docs/decisions/0012-iranian-reporting-week.md | 18 ++++++++++++ 10 files changed, 80 insertions(+), 45 deletions(-) delete mode 100644 docs/decisions/0008-production-write-and-analytics-freshness.md create mode 100644 docs/decisions/0011-lazy-merchant-analytics-cache.md create mode 100644 docs/decisions/0012-iranian-reporting-week.md diff --git a/CONTEXT.md b/CONTEXT.md index 2c03fb3..2ac5f77 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -44,6 +44,10 @@ A deterministic numerical measure with a definition, source fields, filters, per A rebuildable summary derived from normalized source-of-truth records to answer an approved metric efficiently. A projection is not independent business truth. +## Analytics Revision + +A monotonically increasing merchant or cohort version changed when a related payment fact is inserted. Cached projections record the revision they represent; a revision mismatch makes them stale and triggers lazy recomputation on the next relevant dashboard request. + ## Peer Cohort An approved group of other merchants used for anonymized comparison. The active merchant is excluded, and a benchmark is suppressed unless at least 10 peers and 1,000 peer sessions contribute. diff --git a/README.md b/README.md index 1d6a8fc..f19cb3e 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,6 @@ One repository may expose several small `cmd/` entry points while sharing domain - HTTP API - dataset importer - database migration command -- analytics refresh/background job, only if the approved spec requires one +- on-demand analytics calculation within the HTTP API, persisted as merchant-scoped PostgreSQL projections The exact package layout is deferred until the Go architecture and DI decisions are approved. diff --git a/docs/DATA-FOUNDATION-SPEC.md b/docs/DATA-FOUNDATION-SPEC.md index 5a117f6..9fbed3b 100644 --- a/docs/DATA-FOUNDATION-SPEC.md +++ b/docs/DATA-FOUNDATION-SPEC.md @@ -125,6 +125,7 @@ erDiagram text persian_season int iso_weekday date reporting_week_start + date reporting_week_end } MERCHANT_DAY_SUMMARY { bigint merchant_id FK @@ -150,17 +151,20 @@ This contract applies only to the one-time challenge-data bootstrap and explicit 5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships. 6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete. 7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials. -8. Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh. -9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract. +8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard projections; first relevant access owns lazy calculation. +9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions invalidate affected lazy analytics projections. -## Production write and analytics freshness contract +## Production write and lazy analytics-cache contract - ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema. - This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow. -- Live rows are distinguishable from bootstrap rows and include stable creation/update metadata for incremental analytics. Bootstrap-only import provenance is not forged for live rows. -- Incremental analytics projections refresh within 15 minutes of an eligible source change. -- A nightly idempotent reconciliation checks/recomputes the approved authoritative window and detects drift. -- Failed refreshes never advance the API/UI freshness watermark. +- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract. +- The first relevant dashboard request calculates typed merchant analytics projections and persists them in PostgreSQL. Later requests reuse projections for the same metric-definition and source revisions. +- Every inserted related payment fact atomically advances the affected merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers. +- A request with missing or stale projections obtains a merchant-scoped PostgreSQL lock, recalculates once, publishes atomically, and records the exact source/cohort and metric-definition revisions represented. Concurrent requests reuse the completed result rather than duplicate expensive queries. +- Invalidation marks cached results stale through revision mismatch; it does not synchronously recalculate or delete the last successful snapshot. +- Failed calculations never mark a revision current. The API may expose the prior snapshot only with an explicit stale warning and last-calculated metadata. +- No scheduled analytics refresh worker, nightly reconciliation job, Redis, or time-based expiry is required for the static MVP dataset. ## Synthetic merchant/account rules @@ -183,8 +187,11 @@ This contract applies only to the one-time challenge-data bootstrap and explicit - Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics. - Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages. - Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts. -- Incremental projections become queryable within 15 minutes of eligible live writes, and nightly reconciliation detects/corrects drift without double counting. +- First access creates reusable merchant-scoped projections; a second equivalent request reads the current cache without repeating source-fact aggregation. +- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically. +- Concurrent stale-cache requests do not duplicate the same merchant calculation, and failed calculation does not claim the new revision is cached. - Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes. +- Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval. ## Approval gates / unresolved decisions @@ -192,7 +199,7 @@ Resolved: use a capability-oriented modular monolith, with lightweight ports/ada Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002. -Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003. +Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003. Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004. @@ -202,10 +209,12 @@ Resolved: peer benchmarks exclude the active merchant and require at least 10 ot Resolved: use normalized source-of-truth entities plus a reporting-date dimension, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. -Resolved: ZarinPal writes future production rows directly to the normalized tables. Our backend owns schema/read/analytics contracts, refreshes projections within 15 minutes, and performs nightly reconciliation. CSV ingestion remains one-time only. See ADR 0008. +Resolved: ZarinPal writes future append-only production rows directly to normalized tables. Analytics are calculated lazily on first relevant dashboard access, stored as typed PostgreSQL projections, and invalidated through merchant/cohort source revisions advanced on insert. No scheduled refresh or Redis is required. See ADR 0011. Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010. +Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `Asia/Tehran`, represented as half-open API/database ranges. See ADR 0012. + Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`. No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, reporting-period boundaries, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec. diff --git a/docs/PHYSICAL-SCHEMA-PROPOSAL.md b/docs/PHYSICAL-SCHEMA-PROPOSAL.md index d8887e7..cd4bfef 100644 --- a/docs/PHYSICAL-SCHEMA-PROPOSAL.md +++ b/docs/PHYSICAL-SCHEMA-PROPOSAL.md @@ -49,7 +49,7 @@ Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Creden `adjusted_fee` uses fixed precision and is labeled as an obfuscated relative value. The physical precision is finalized from exact decimal profiling before migration approval. -Both fact tables distinguish `bootstrap` from `zarinpal_live` origin. Bootstrap rows reference an import run; live rows do not invent one. Live creation/update metadata forms the incremental analytics watermark contract, including status changes to an existing session. +Both fact tables distinguish `bootstrap` from `zarinpal_live` origin. Bootstrap rows reference an import run; live rows do not invent one. Payment facts are append-only for the MVP; inserts advance database-managed analytics revisions. ### Rebuildable analytics projections @@ -59,10 +59,12 @@ Both fact tables distinguish `bootstrap` from `zarinpal_live` origin. Bootstrap | `merchant_day_*_breakdowns` | merchant + date + approved dimension | Only metric-catalogue breakdowns proven useful, such as PSP/failure groups | | `merchant_period_*` | merchant + approved day/week/month/season period | Non-additive or measured expensive metrics only | | `cohort_period_*` | approved cohort + period | Suppressed peer benchmark inputs/outputs with provenance and freshness | +| `merchant_analytics_state` | one row per merchant | current source revision, cached revision, metric-definition version, calculation status/time | +| `cohort_analytics_state` | one row per benchmark cohort/category | current source revision and cached revision for peer projections | Projection names/columns are finalized by the analytics metric catalogue. Avoid a generic key/value metrics table because it weakens types, constraints, and discoverability. -Projection metadata records source watermark, calculation version, calculated time, and status. An incremental refresh runs at least every 15 minutes; a nightly reconciliation checks/rebuilds an authoritative window and advances freshness only after successful atomic publication. +Projection rows record merchant/cohort source revision, metric-definition version, calculated time, and publication status. Inserts atomically increment the affected revision through schema-owned database logic so direct upstream writes cannot bypass invalidation. A dashboard cache miss or revision mismatch acquires a PostgreSQL advisory/row lock, builds typed projections, and advances the cached revision only after atomic publication. No generic JSON metric cache, Redis, scheduled refresh, or time-to-live expiry is required. ## Initial index policy @@ -77,9 +79,10 @@ Projection metadata records source watermark, calculation version, calculated ti - Payment/import provenance uses restrictive foreign keys; no cascading deletion of analytical facts from merchant/profile operations. - Imported source facts are immutable after a completed import. Corrections create a new versioned recovery/import decision. +- Production payment facts are append-only in the MVP. Update/delete semantics require a future ADR and corresponding cache-invalidation rules. - Synthetic display profiles and credentials can change independently from source facts. -- Derived projections may be rebuilt and atomically replaced; freshness/provenance prevents stale data from appearing current. +- Derived projections may be rebuilt and atomically replaced; source and metric-definition revisions prevent stale data from appearing current. ## Production population boundary -After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records directly to the normalized PostgreSQL session/attempt tables. This project owns the schema/migration and analytics contract but not ZarinPal's write workflow. Compatibility, required fields, source origin, and creation/update watermarks are versioned interfaces; recurring CSV ETL is prohibited. +After the one-time CSV bootstrap, ZarinPal—not this project—writes future production payment records directly to the normalized PostgreSQL session/attempt tables. This project owns the schema/migration, insert-triggered analytics-revision, and read contract but not ZarinPal's write workflow. Compatibility, required fields, source origin, and append-only insertion semantics are versioned interfaces; recurring CSV ETL is prohibited. diff --git a/docs/decisions/0003-ent-pgx-data-access.md b/docs/decisions/0003-ent-pgx-data-access.md index 58fe57e..7946696 100644 --- a/docs/decisions/0003-ent-pgx-data-access.md +++ b/docs/decisions/0003-ent-pgx-data-access.md @@ -13,7 +13,7 @@ Use Ent for application persistence, relationship modeling, and type-safe access Use pgx directly for the importer’s PostgreSQL staging boundary. Feed `CopyFrom` with a streaming `CopyFromSource` (or equivalent bounded batches) so decompressed CSV rows are not accumulated in memory. Validate and promote staged data through explicit transactions/steps described by the import spec; bulk loading must not bypass reconciliation or data-quality rules. -Use versioned SQL migrations generated/planned from the reviewed schema and committed to Git. Apply them through a dedicated migration command or deployment step. Ent automatic schema creation is allowed only in disposable development/test contexts if the later implementation spec permits it; it must never run as part of production application startup. +Use Atlas, Ent's supported migration engine, to generate/plan versioned SQL migrations from the reviewed Ent schema. Review and commit the SQL to Git, then apply it through a dedicated deployment migration step. Atlas is build/deployment tooling, not a runtime service. Ent automatic schema creation is allowed only in disposable development/test contexts if the later implementation spec permits it; it must never run as part of production application startup. ## Consequences @@ -21,10 +21,10 @@ Use versioned SQL migrations generated/planned from the reviewed schema and comm - Engineers must still inspect generated SQL, query plans, indexes, transaction boundaries, and possible N+1 behavior; the ORM is not a performance or correctness boundary. - The importer owns a deliberate lower-level pgx path and cannot reuse ordinary Ent writes for convenience. - Both Ent and pgx require compatible pool/transaction ownership and context propagation; adapters must not hide unbounded work. -- Migration generation tooling and the migration runner remain to be selected in the implementation spec. Versioned migration files—not runtime diffs—are the production contract. +- Atlas-generated versioned migration files—not runtime diffs—are the production contract. CI validates migration integrity and drift before deployment. ## References - Ent versioned migrations: +- Atlas versioned migrations: - pgx v5 `CopyFrom`: - diff --git a/docs/decisions/0007-calendar-aware-analytics-projections.md b/docs/decisions/0007-calendar-aware-analytics-projections.md index 0fe24ff..34844c3 100644 --- a/docs/decisions/0007-calendar-aware-analytics-projections.md +++ b/docs/decisions/0007-calendar-aware-analytics-projections.md @@ -26,5 +26,5 @@ Keep normalized fact tables unpartitioned initially. Add targeted indexes from a - The normalized ERD remains semantically correct while dashboards read smaller, purpose-built projections. - Daily components support many calendar groupings without precomputing every possible period. - Explicit reporting-period definitions and metric provenance make future AI observations traceable. -- Projection refresh, failure, stale-state visibility, and reconciliation tests become required operational behavior. +- Lazy calculation, revision invalidation, concurrency deduplication, failure, and stale-state visibility tests become required behavior. - Schema/index/projection details remain subject to data profiling and approved metric/API requirements. diff --git a/docs/decisions/0008-production-write-and-analytics-freshness.md b/docs/decisions/0008-production-write-and-analytics-freshness.md deleted file mode 100644 index 3f0d7c3..0000000 --- a/docs/decisions/0008-production-write-and-analytics-freshness.md +++ /dev/null @@ -1,25 +0,0 @@ -# ADR 0008: ZarinPal owns live writes; analytics refresh within 15 minutes - -- Status: Accepted -- Date: 2026-08-20 - -## Context - -The challenge archive is a one-time bootstrap source. In production, ZarinPal already creates payment data and will write future records directly to the normalized PostgreSQL schema. Building a recurring CSV importer, polling source, or competing payment-write workflow would duplicate upstream responsibility. Derived dashboard projections still need bounded freshness and correction when live source rows arrive or change status. - -## Decision - -ZarinPal is the authoritative producer of live `payment_sessions` and `payment_attempts`. This backend owns the normalized schema contract, versioned migrations, analytical reads/projections, and tenant-safe APIs, but not the business workflow that produces live payment records. - -Fact rows identify `bootstrap` versus `zarinpal_live` origin. Bootstrap rows reference import provenance; live rows do not fabricate import runs. The live-write contract supplies stable creation/update metadata so an idempotent high-watermark process can include new rows and status updates. - -Refresh affected analytics projections within 15 minutes. Run a nightly idempotent reconciliation over an approved authoritative window to detect and correct missed/changed rows. Publish projections atomically and advance the visible freshness watermark only after success. - -## Consequences - -- No recurring CSV ingestion service, file watcher, or source polling is built. -- Schema migrations require compatibility coordination with ZarinPal's writer. -- Analytics refresh is separate from ingestion and requires scheduling, checkpoints, idempotency, metrics, and stale-state UX. -- If upstream cannot provide reliable update metadata, revise the refresh algorithm/window before implementation rather than silently missing mutations. -- The static hackathon dataset builds projections once after bootstrap; production scheduling remains testable with synthetic live writes. - diff --git a/docs/decisions/0010-limit-reporting-periods.md b/docs/decisions/0010-limit-reporting-periods.md index 8caaa51..3488542 100644 --- a/docs/decisions/0010-limit-reporting-periods.md +++ b/docs/decisions/0010-limit-reporting-periods.md @@ -11,7 +11,7 @@ Earlier drafts included named Iranian holidays and analytical event windows. Tha Limit reporting periods to Tehran-local daily, weekly, Jalali-month, and Persian-season views. Remove named holidays, Nowruz-specific periods, public-holiday calendars, and generic event-window reporting from the current product and data model. -The exact weekly boundary remains an explicit reporting-period decision; it must not be inferred differently by the database, API, and frontend. +The reporting week follows the separately approved Saturday-through-Friday boundary in ADR 0012 and must not be inferred differently by the database, API, and frontend. ## Consequences diff --git a/docs/decisions/0011-lazy-merchant-analytics-cache.md b/docs/decisions/0011-lazy-merchant-analytics-cache.md new file mode 100644 index 0000000..ba022c0 --- /dev/null +++ b/docs/decisions/0011-lazy-merchant-analytics-cache.md @@ -0,0 +1,26 @@ +# ADR 0011: Calculate and cache merchant analytics lazily + +- Status: Accepted +- Date: 2026-08-20 + +## Context + +The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before any merchant requests the data. The production boundary may later receive new append-only rows written directly by ZarinPal, bypassing Go request handlers, so any invalidation contract must also work at the database boundary. + +## Decision + +Calculate a merchant's approved analytics projections on the first relevant dashboard request and persist typed projections in PostgreSQL. Reuse them while their merchant source revision, relevant cohort revision, and metric-definition version remain current. + +Schema-owned database logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts that affect peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes without a refresh worker or Redis. + +On a cache miss or revision mismatch, the backend obtains a merchant-scoped PostgreSQL advisory/row lock, computes the projections once, publishes them atomically, and then advances the cached revision. Concurrent requests wait for or reuse that result. A failed calculation does not advance the cached revision; an older snapshot may be returned only when clearly marked stale. + +Payment facts are append-only for this MVP. Cache invalidation for updates or hard deletion is out of scope and requires a future contract change. + +## Consequences + +- The static dataset is aggregated at most once per merchant and metric-definition revision when requested. +- No scheduled analytics worker, nightly reconciliation, Redis, or time-based cache expiry is required. +- Cache correctness is version-based, not age-based. +- Cohort revision tracking prevents a new merchant fact from silently leaving peer comparisons current for other merchants. +- Trigger/revision behavior, concurrency deduplication, failure recovery, and traceability require integration tests. diff --git a/docs/decisions/0012-iranian-reporting-week.md b/docs/decisions/0012-iranian-reporting-week.md new file mode 100644 index 0000000..d70d8b0 --- /dev/null +++ b/docs/decisions/0012-iranian-reporting-week.md @@ -0,0 +1,18 @@ +# ADR 0012: Use Saturday-through-Friday reporting weeks + +- Status: Accepted +- Date: 2026-08-20 + +## Context + +Weekly analytics require one boundary shared by PostgreSQL grouping, API filters, cache keys, comparisons, and Persian UI labels. + +## Decision + +Use the Iranian week in `Asia/Tehran`: Saturday is the first day and Friday is the last day. Store the Tehran-local Saturday date as the stable week key. API and query ranges use a half-open interval from Saturday `00:00` inclusive to the following Saturday `00:00` exclusive. + +## Consequences + +- Week boundaries remain correct and unambiguous across daylight/timezone handling. +- A week crossing a Jalali month or year remains identified by its starting Saturday date. +- Previous-period comparison uses the immediately preceding Saturday-through-Friday interval.