starcoach-hackathon/zarinpal-analytics-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: move lazy analytics cache to redis

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 16:28:29 +03:30
parent 680596fe73
commit be3cb9f36b
7 changed files with 72 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/DATA-FOUNDATION-SPEC.md
View File

@@ -15,7 +15,7 @@ Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
- Rebuildable daily base summaries and metric-specific period projections for approved analytics
- PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
@@ -40,8 +40,6 @@ erDiagram
MERCHANT ||--o{ PAYMENT_SESSION : receives
TERMINAL ||--o{ PAYMENT_SESSION : handles
PAYMENT_SESSION ||--|{ PAYMENT_ATTEMPT : contains
CALENDAR_DATE ||--o{ MERCHANT_DAY_SUMMARY : groups
MERCHANT ||--o{ MERCHANT_DAY_SUMMARY : summarizes
IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
@@ -127,18 +125,9 @@ erDiagram
date reporting_week_start
date reporting_week_end
}
MERCHANT_DAY_SUMMARY {
bigint merchant_id FK
date local_date FK
bigint session_count
bigint attempt_count
bigint successful_count
bigint amount_sum
bigint source_import_run_id FK
}
```
This ERD is a logical baseline, not an approved physical schema. In particular, source key types, card-key retention, analytics projections/materialized views, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.
This ERD is a logical baseline, not an approved physical schema. In particular, source key types, cache/invalidation state, and whether session-level fields can conflict across retries must be resolved from a profiling report before migrations are approved.
## Import contract
@@ -151,20 +140,23 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
5. Validate uniqueness of `(session_key, try_seq)`, referential consistency, parseability, null rates, ranges, timestamp ordering, and merchant/terminal/category relationships.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard projections; first relevant access owns lazy calculation.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions invalidate affected lazy analytics projections.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.
## Production write and lazy analytics-cache contract
- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
- The first relevant dashboard request calculates typed merchant analytics projections and persists them in PostgreSQL. Later requests reuse projections for the same metric-definition and source revisions.
- Every inserted related payment fact atomically advances the affected merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- A request with missing or stale projections obtains a merchant-scoped PostgreSQL lock, recalculates once, publishes atomically, and records the exact source/cohort and metric-definition revisions represented. Concurrent requests reuse the completed result rather than duplicate expensive queries.
- Invalidation marks cached results stale through revision mismatch; it does not synchronously recalculate or delete the last successful snapshot.
- Failed calculations never mark a revision current. The API may expose the prior snapshot only with an explicit stale warning and last-calculated metadata.
- No scheduled analytics refresh worker, nightly reconciliation job, Redis, or time-based expiry is required for the static MVP dataset.
- The first relevant dashboard access starts all approved analytics-section jobs. A bounded on-demand worker pool calculates independent sections in parallel rather than running a scheduled refresh.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries use a configurable high TTL.
- The authenticated SSE stream emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
- Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
- Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
- Failed sections emit a typed failure event and remain retryable without marking the section ready. Other successful sections remain available.
- Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.
## Synthetic merchant/account rules
@@ -187,9 +179,12 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
- Daily, weekly, Jalali-month, and Persian-season analytics reconcile to deterministic source queries for additive metrics.
- Rates are recomputed from their numerator and denominator over the requested period rather than averaged from daily percentages.
- Non-additive metrics such as unique payer-token counts are never produced by summing daily distinct counts.
- First access creates reusable merchant-scoped projections; a second equivalent request reads the current cache without repeating source-fact aggregation.
- First access starts every core analytics section; each section is delivered through authenticated SSE as soon as it is ready.
- A second equivalent request reads current high-TTL Redis entries without repeating source-fact aggregation.
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
- Concurrent stale-cache requests do not duplicate the same merchant calculation, and failed calculation does not claim the new revision is cached.
- Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
- One failed section does not hide ready sections and does not create a ready cache entry.
- Redis loss produces a cold-cache calculation rather than loss of source data.
- Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes.
- Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval.
@@ -207,9 +202,9 @@ Resolved: retain `payer_card_key` unchanged as a restricted, merchant-scoped opa
Resolved: peer benchmarks exclude the active merchant and require at least 10 other merchants plus 1,000 combined peer sessions in the approved cohort/window. Otherwise the benchmark is suppressed as insufficient data; the merchant's own analytics remain available. See ADR 0006.
Resolved: use normalized source-of-truth entities plus a reporting-date dimension, rebuildable daily base summaries, and metric-specific projections for non-additive or measured high-cost period analytics. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. Analytics are calculated lazily on first relevant dashboard access, stored as typed PostgreSQL projections, and invalidated through merchant/cohort source revisions advanced on insert. No scheduled refresh or Redis is required. See ADR 0011.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. First dashboard access starts all section jobs in bounded parallel, Redis caches each typed result with a high TTL, and authenticated SSE reveals sections as they become ready. PostgreSQL merchant/cohort revisions advanced on insert provide logical invalidation. See ADR 0011.
Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.
@@ -217,4 +212,4 @@ Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `A
Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`.
No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, reporting-period boundaries, projection shapes, and migration details still require profiling evidence and acceptance in the implementation-ready revision of this spec.
No broad architecture approval gates remain for this capability. Physical columns, indexes, constraints, Redis TTL/limits, SSE delivery details, analytics-section contracts, and migration details still require acceptance in the implementation-ready specs.

21
docs/PHYSICAL-SCHEMA-PROPOSAL.md
View File

@@ -10,7 +10,7 @@
- Preserve source identifiers as text and use internal surrogate keys for joins.
- Keep payment session and payment attempt grains separate.
- Use UTC `timestamptz` for canonical instants and `bigint` for Rial amounts.
- Keep normalized facts as source of truth; analytics projections are rebuildable.
- Keep normalized PostgreSQL facts as source of truth; Redis analytics results are disposable and rebuildable.
- No initial table partitioning. Add indexes from approved access paths and measured query plans.
- The CSV importer is a one-time administrative bootstrap, not a production ingestion service.
@@ -51,20 +51,16 @@ Synthetic profile fields carry an explicit `profile_is_synthetic` marker. Creden
Both fact tables distinguish `bootstrap` from `zarinpal_live` origin. Bootstrap rows reference an import run; live rows do not invent one. Payment facts are append-only for the MVP; inserts advance database-managed analytics revisions.
### Rebuildable analytics projections
### Analytics invalidation state
| Table family | Grain | Purpose |
| Table | Grain | Purpose |
|---|---|---|
| `merchant_day_metrics` | merchant + Tehran local date | Additive counts/sums and metric numerators/denominators |
| `merchant_day_*_breakdowns` | merchant + date + approved dimension | Only metric-catalogue breakdowns proven useful, such as PSP/failure groups |
| `merchant_period_*` | merchant + approved day/week/month/season period | Non-additive or measured expensive metrics only |
| `cohort_period_*` | approved cohort + period | Suppressed peer benchmark inputs/outputs with provenance and freshness |
| `merchant_analytics_state` | one row per merchant | current source revision, cached revision, metric-definition version, calculation status/time |
| `cohort_analytics_state` | one row per benchmark cohort/category | current source revision and cached revision for peer projections |
| `merchant_analytics_revisions` | one row per merchant | monotonically increasing source revision advanced by related fact inserts |
| `cohort_analytics_revisions` | one row per benchmark cohort/category | monotonically increasing peer-source revision advanced by relevant inserts |
Projection names/columns are finalized by the analytics metric catalogue. Avoid a generic key/value metrics table because it weakens types, constraints, and discoverability.
No calculated dashboard result is stored in PostgreSQL. Analytics queries and Redis value contracts are finalized by the metric catalogue and API spec.
Projection rows record merchant/cohort source revision, metric-definition version, calculated time, and publication status. Inserts atomically increment the affected revision through schema-owned database logic so direct upstream writes cannot bypass invalidation. A dashboard cache miss or revision mismatch acquires a PostgreSQL advisory/row lock, builds typed projections, and advances the cached revision only after atomic publication. No generic JSON metric cache, Redis, scheduled refresh, or time-to-live expiry is required.
Inserts atomically increment affected revision rows through schema-owned database logic so direct upstream writes cannot bypass invalidation. Redis keys include merchant/cohort revisions, metric-definition version, and analytics-section id. Old revision keys expire under a configurable high TTL; invalidation does not scan or delete arbitrary keys. Redis values follow typed, versioned response schemas and contain aggregates only.
## Initial index policy
@@ -72,7 +68,6 @@ Projection rows record merchant/cohort source revision, metric-definition versio
- `payment_sessions (merchant_id, created_at DESC)` supports merchant period queries.
- `payment_sessions (terminal_id, created_at DESC)` supports terminal drill-down.
- `payment_attempts (payment_session_id, try_seq)` is unique and supports session traceability.
- Projection primary keys start with merchant/cohort and period.
- Additional status, PSP, bank, payer-token, or timestamp indexes require an approved API query and `EXPLAIN (ANALYZE, BUFFERS)` evidence; do not index every CSV column.
## Deletion and mutation policy
@@ -81,7 +76,7 @@ Projection rows record merchant/cohort source revision, metric-definition versio
- Imported source facts are immutable after a completed import. Corrections create a new versioned recovery/import decision.
- Production payment facts are append-only in the MVP. Update/delete semantics require a future ADR and corresponding cache-invalidation rules.
- Synthetic display profiles and credentials can change independently from source facts.
- Derived projections may be rebuilt and atomically replaced; source and metric-definition revisions prevent stale data from appearing current.
- Redis analytics entries are disposable and rebuilt on a cache miss. Source/cohort and metric-definition revisions prevent stale keys from appearing current.
## Production population boundary

12
docs/decisions/0007-calendar-aware-analytics-projections.md
View File

@@ -15,16 +15,16 @@ Keep normalized business/source-of-truth relations: categories, merchants, merch
Add a reporting-date model containing Gregorian and Jalali date attributes, explicit reporting-week boundaries, Persian months/seasons, and weekdays.
Build idempotent, rebuildable daily base projections containing additive components such as session count, attempt count, successful count, amount sum, and fee sum as approved by the metric catalogue. Derive week/month/season totals by grouping these components through the reporting-date model and recompute ratios from their numerators and denominators.
Build idempotent, rebuildable analytics-section queries containing additive components such as session count, attempt count, successful count, amount sum, and fee sum as approved by the metric catalogue. Derive week/month/season totals through the reporting-date model and recompute ratios from their numerators and denominators.
Create separate metric-specific period projections only for non-additive metrics or query paths proven expensive—for example, exact unique payer-token counts cannot be obtained by summing daily distinct counts. Projection rows record source import/version and freshness.
Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under source/cohort and metric-definition revisions.
Keep normalized fact tables unpartitioned initially. Add targeted indexes from approved API access paths and verified query plans. Introduce PostgreSQL partitioning only through a future ADR supported by measured table growth, query latency, vacuum/maintenance behavior, or retention operations.
## Consequences
- The normalized ERD remains semantically correct while dashboards read smaller, purpose-built projections.
- Daily components support many calendar groupings without precomputing every possible period.
- The normalized ERD remains semantically correct while dashboards reuse disposable Redis results.
- Section calculations support approved period groupings without storing calculated facts as PostgreSQL business truth.
- Explicit reporting-period definitions and metric provenance make future AI observations traceable.
- Lazy calculation, revision invalidation, concurrency deduplication, failure, and stale-state visibility tests become required behavior.
- Schema/index/projection details remain subject to data profiling and approved metric/API requirements.
- Lazy parallel calculation, Redis revision keys, job deduplication, partial failure, and SSE visibility tests become required behavior.
- Schema/index/cache details remain subject to data profiling and approved metric/API requirements.

26
docs/decisions/0011-lazy-merchant-analytics-cache.md
View File

@@ -1,26 +0,0 @@
# ADR 0011: Calculate and cache merchant analytics lazily
- Status: Accepted
- Date: 2026-08-20
## Context
The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before any merchant requests the data. The production boundary may later receive new append-only rows written directly by ZarinPal, bypassing Go request handlers, so any invalidation contract must also work at the database boundary.
## Decision
Calculate a merchant's approved analytics projections on the first relevant dashboard request and persist typed projections in PostgreSQL. Reuse them while their merchant source revision, relevant cohort revision, and metric-definition version remain current.
Schema-owned database logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts that affect peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes without a refresh worker or Redis.
On a cache miss or revision mismatch, the backend obtains a merchant-scoped PostgreSQL advisory/row lock, computes the projections once, publishes them atomically, and then advances the cached revision. Concurrent requests wait for or reuse that result. A failed calculation does not advance the cached revision; an older snapshot may be returned only when clearly marked stale.
Payment facts are append-only for this MVP. Cache invalidation for updates or hard deletion is out of scope and requires a future contract change.
## Consequences
- The static dataset is aggregated at most once per merchant and metric-definition revision when requested.
- No scheduled analytics worker, nightly reconciliation, Redis, or time-based cache expiry is required.
- Cache correctness is version-based, not age-based.
- Cohort revision tracking prevents a new merchant fact from silently leaving peer comparisons current for other merchants.
- Trigger/revision behavior, concurrency deduplication, failure recovery, and traceability require integration tests.

29
docs/decisions/0011-lazy-redis-analytics-cache.md Normal file
View File

@@ -0,0 +1,29 @@
# ADR 0011: Calculate analytics sections lazily and cache them in Redis
- Status: Accepted
- Date: 2026-08-20
## Context
The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Future append-only rows may be written directly by ZarinPal, bypassing Go request handlers.
## Decision
On the first relevant dashboard access, enqueue every approved analytics section for the authenticated merchant. A bounded on-demand worker pool calculates independent sections in parallel. This worker is event-driven, not scheduled.
Cache each completed typed section in Redis using a key derived from the merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Use a configurable high TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. Old-revision keys expire naturally and are never treated as current.
Stream progress and section availability through an authenticated, merchant-scoped SSE connection. Emit stable event ids and typed events for section started, section ready, section failed, overall complete, and heartbeat. Support safe reconnect without exposing another merchant’s jobs or values. One section failure does not suppress sections that succeeded.
Schema-owned PostgreSQL logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts affecting peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes. Payment facts are append-only for the MVP; invalidation for update/delete remains out of scope.
Redis is a private disposable cache and coordination system, not a source of truth. PostgreSQL remains authoritative. Cached values contain approved aggregates only and exclude raw rows, payer-card keys, credentials, session tokens, and cross-tenant details.
## Consequences
- All dashboard analytics begin together, while the UI reveals each section independently as it completes.
- Redis loss or key expiry causes safe lazy recomputation; Redis is excluded from backup/restore requirements.
- No scheduled refresh or time-based freshness claim is required.
- The deployment includes private Redis and on-demand analytics-worker services.
- Bounded global/per-merchant concurrency, job timeouts, retry limits, SSE connection limits, and cache TTL still require explicit values.
- Trigger/revision behavior, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests.