starcoach-hackathon/zarinpal-analytics-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: move lazy analytics cache to redis

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 16:28:29 +03:30
parent 680596fe73
commit be3cb9f36b
7 changed files with 72 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/DATA-FOUNDATION-SPEC.md
View File

@@ -15,7 +15,7 @@ Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
- Rebuildable daily base summaries and metric-specific period projections for approved analytics
- PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
@@ -40,8 +40,6 @@ erDiagram
MERCHANT ||--o{ PAYMENT_SESSION : receives
TERMINAL ||--o{ PAYMENT_SESSION : handles
PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups
MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes
IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
@@ -127,18 +125,9 @@ erDiagram
date reporting_week_start
date reporting_week_end
}
MERCHANT_DAY_SUMMARY {
bigint merchant_id FK
date local_date FK
bigint session_count
bigint attempt_count
bigint successful_count
bigint amount_sum
bigint source_import_run_id FK
}
```
This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.
This ERD is a logical baseline, not an approved physical schema. In particular, source key types, cache/invalidation state, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.
## Import contract
@@ -151,20 +140,23 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard projections; first relevant access owns lazy calculation.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions invalidate affected lazy analytics projections.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.
## Production write and lazy analytics-cache contract
- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
- The first relevant dashboard request calculates typed merchant analytics projections and persists them in PostgreSQL. Later requests reuse projections for the same metric-definition and source revisions.
- Every inserted related payment fact atomically advances the affected merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- A request with missing or stale projections obtains a merchant-scoped PostgreSQL lock, recalculates once, publishes atomically, and records the exact source/cohort and metric-definition revisions represented. Concurrent requests reuse the completed result rather than duplicate expensive queries.
- Invalidation marks cached results stale through revision mismatch; it does not synchronously recalculate or delete the last successful snapshot.
- Failed calculations never mark a revision current. The API may expose the prior snapshot only with an explicit stale warning and last-calculated metadata.
- No scheduled analytics refresh worker, nightly reconciliation job, Redis, or time-based expiry is required for the static MVP dataset.
- The first relevant dashboard access starts all approved analytics-section jobs. A bounded on-demand worker pool calculates independent sections in parallel rather than running a scheduled refresh.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries use a configurable high TTL.
- The authenticated SSE stream emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
- Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
- Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
- Failed sections emit a typed failure event and remain retryable without marking the section ready. Other successful sections remain available.
- Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.
## Synthetic merchant/account rules
@@ -187,9 +179,12 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
- Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics.
- Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
- Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
- First access creates reusable merchant-scoped projections; a second equivalent request reads the current cache without repeating source-fact aggregation.
- First access starts every core analytics section; each section is delivered through authenticated SSE as soon as it is ready.
- A second equivalent request reads current high-TTL Redis entries without repeating source-fact aggregation.
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
- Concurrent stale-cache requests do not duplicate the same merchant calculation, and failed calculation does not claim the new revision is cached.
- Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
- One failed section does not hide ready sections and does not create a ready cache entry.
- Redis loss produces a cold-cache calculation rather than loss of source data.
- Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes.
- Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval.
@@ -207,9 +202,9 @@ Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opa
Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006.
Resolved: use normalized source-of-truth entities plus a reporting-date dimension, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. Analytics are calculated lazily on first relevant dashboard access, stored as typed PostgreSQL projections, and invalidated through merchant/cohort source revisions advanced on insert. No scheduled refresh or Redis is required. See ADR 0011.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. First dashboard access starts all section jobs in bounded parallel, Redis caches each typed result with a high TTL, and authenticated SSE reveals sections as they become ready. PostgreSQL merchant/cohort revisions advanced on insert provide logical invalidation. See ADR 0011.
Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.
@@ -217,4 +212,4 @@ Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `A
Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`.
No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, reporting-period boundaries, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.
No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, Redis TTL/limits, SSE delivery details, analytics-section contracts, and migration details still require acceptance in the implementation-ready specs.