starcoach-hackathon/zarinpal-analytics-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: adopt lazy analytics caching

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 16:17:27 +03:30
parent 7c8afeaa4b
commit 680596fe73
10 changed files with 80 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
docs/DATA-FOUNDATION-SPEC.md
View File

@@ -125,6 +125,7 @@ erDiagram
text persian_season
int iso_weekday
date reporting_week_start
date reporting_week_end
}
MERCHANT_DAY_SUMMARY {
bigint merchant_id FK
@@ -150,17 +151,20 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved calendar range, then build affected analytics projections idempotently after source promotion. Projection failure must not make an incomplete projection appear fresh.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Analytics freshness over those future rows has a separate read/refresh contract.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard projections; first relevant access owns lazy calculation.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions invalidate affected lazy analytics projections.
## Production write and analytics freshness contract
## Production write and lazy analytics-cache contract
- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
- Live rows are distinguishable from bootstrap rows and include stable creation/update metadata for incremental analytics. Bootstrap-only import provenance is not forged for live rows.
- Incremental analytics projections refresh within 15 minutes of an eligible source change.
- A nightly idempotent reconciliation checks/recomputes the approved authoritative window and detects drift.
- Failed refreshes never advance the API/UI freshness watermark.
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
- The first relevant dashboard request calculates typed merchant analytics projections and persists them in PostgreSQL. Later requests reuse projections for the same metric-definition and source revisions.
- Every inserted related payment fact atomically advances the affected merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- A request with missing or stale projections obtains a merchant-scoped PostgreSQL lock, recalculates once, publishes atomically, and records the exact source/cohort and metric-definition revisions represented. Concurrent requests reuse the completed result rather than duplicate expensive queries.
- Invalidation marks cached results stale through revision mismatch; it does not synchronously recalculate or delete the last successful snapshot.
- Failed calculations never mark a revision current. The API may expose the prior snapshot only with an explicit stale warning and last-calculated metadata.
- No scheduled analytics refresh worker, nightly reconciliation job, Redis, or time-based expiry is required for the static MVP dataset.
## Synthetic merchant/account rules
@@ -183,8 +187,11 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
- Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics.
- Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
- Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
- Incremental projections become queryable within 15 minutes of eligible live writes, and nightly reconciliation detects/corrects drift without double counting.
- First access creates reusable merchant-scoped projections; a second equivalent request reads the current cache without repeating source-fact aggregation.
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
- Concurrent stale-cache requests do not duplicate the same merchant calculation, and failed calculation does not claim the new revision is cached.
- Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes.
- Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval.
## Approval gates / unresolved decisions
@@ -192,7 +199,7 @@ Resolved: use a capability-oriented modular monolith, with lightweight ports/ada
Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002.
Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003.
Resolved: use Ent for application persistence/modeling, pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging, and Atlas for reviewed versioned SQL migrations. Production startup never performs automatic schema changes. See ADR 0003.
Resolved: interpret timezone-naive source timestamps as `Asia/Tehran`, normalize canonical timestamps to UTC, expose ISO-8601 timestamps through APIs, and retain the immutable source/import provenance needed to reproduce the conversion. See ADR 0004.
@@ -202,10 +209,12 @@ Resolved: peer benchmarks exclude the active merchant and require at least 10 ot
Resolved: use normalized source-of-truth entities plus a reporting-date dimension, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: ZarinPal writes future production rows directly to the normalized tables. Our backend owns schema/read/analytics contracts, refreshes projections within 15 minutes, and performs nightly reconciliation. CSV ingestion remains one-time only. See ADR 0008.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. Analytics are calculated lazily on first relevant dashboard access, stored as typed PostgreSQL projections, and invalidated through merchant/cohort source revisions advanced on insert. No scheduled refresh or Redis is required. See ADR 0011.
Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.
Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `Asia/Tehran`, represented as half-open API/database ranges. See ADR 0012.
Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`.
No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, reporting-period boundaries, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.