docs(cache): remove static dataset invalidation
This commit is contained in:
@@ -17,7 +17,7 @@ Add a reporting-date model containing Gregorian and Jalali date attributes, expl
|
||||
|
||||
Build idempotent, rebuildable analytics-section queries containing additive components such as session count, attempt count, successful count, amount sum, and fee sum as approved by the metric catalogue. Derive week/month/season totals through the reporting-date model and recompute ratios from their numerators and denominators.
|
||||
|
||||
Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under source/cohort and metric-definition revisions.
|
||||
Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under the immutable completed-import snapshot and metric-definition version.
|
||||
|
||||
Keep normalized fact tables unpartitioned initially. Add targeted indexes from approved API access paths and verified query plans. Introduce PostgreSQL partitioning only through a future ADR supported by measured table growth, query latency, vacuum/maintenance behavior, or retention operations.
|
||||
|
||||
@@ -26,5 +26,5 @@ Keep normalized fact tables unpartitioned initially. Add targeted indexes from a
|
||||
- The normalized ERD remains semantically correct while dashboards reuse disposable Redis results.
|
||||
- Section calculations support approved period groupings without storing calculated facts as PostgreSQL business truth.
|
||||
- Explicit reporting-period definitions and metric provenance make future AI observations traceable.
|
||||
- Lazy parallel calculation, Redis revision keys, job deduplication, partial failure, and SSE visibility tests become required behavior.
|
||||
- Lazy parallel calculation, dataset-versioned Redis keys, job deduplication, partial failure, and SSE visibility tests become required behavior.
|
||||
- Schema/index/cache details remain subject to data profiling and approved metric/API requirements.
|
||||
|
||||
@@ -5,15 +5,15 @@
|
||||
|
||||
## Context
|
||||
|
||||
The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Future append-only rows may be written directly by ZarinPal, bypassing Go request handlers.
|
||||
The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Production ZarinPal write integration is outside the MVP.
|
||||
|
||||
## Decision
|
||||
|
||||
On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/source/cohort/metric-definition revision. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled.
|
||||
On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/dataset-snapshot/metric-definition version. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled.
|
||||
|
||||
Use `github.com/hibiken/asynq` pinned to `v0.26.0` as the Redis-backed queue/worker library. Asynq is explicitly pre-`v1.0.0`, and its public API may change without a major-version increment. Do not float the dependency or upgrade it without reviewing release notes, API compatibility, Redis compatibility, retry/deduplication behavior, and integration tests.
|
||||
|
||||
Cache each completed typed section in Redis using a key derived from the merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. Old-revision keys expire naturally and are never treated as current.
|
||||
Cache each completed typed section in Redis using a key derived from the sole completed import-run snapshot, merchant id, period, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. A future explicitly replaced dataset would require a newly specified namespace-selection rule.
|
||||
|
||||
Expose a separate authenticated `GET` SSE endpoint for the run. Stream stable event ids and typed events for section started, section ready, section failed, overall complete, and heartbeat. Support safe reconnect without exposing another merchant’s jobs or values. One section failure does not suppress sections that succeeded.
|
||||
|
||||
@@ -27,11 +27,11 @@ Apply authenticated rate limits independently from job deduplication:
|
||||
- manual retry: 3 requests/10 minutes in independent per-user and per-merchant buckets, burst 1;
|
||||
- SSE: 2 concurrent connections per user and 4 per merchant.
|
||||
|
||||
Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current revisions and is idempotent; it cannot restart ready siblings.
|
||||
Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current dataset snapshot and is idempotent; it cannot restart ready siblings.
|
||||
|
||||
Send SSE heartbeat events every 15 seconds. Retain run events for one hour for authorized `Last-Event-ID` replay. After replay retention expires, return/provide the current run snapshot and continue from new events rather than replaying another merchant’s or an unbounded event history.
|
||||
|
||||
Schema-owned PostgreSQL logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts affecting peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes. Payment facts are append-only for the MVP; invalidation for update/delete remains out of scope.
|
||||
Payment facts are immutable during the MVP, so PostgreSQL contains no cache-revision tables or invalidation triggers. Explicit dataset replacement is an administrative recovery workflow: it selects a different completed import run as the active snapshot and therefore creates a different Redis namespace. Any future live-write integration must define insert/update/delete invalidation in a separately approved specification.
|
||||
|
||||
Redis is a private disposable cache and coordination system, not a source of truth. PostgreSQL remains authoritative. Cached values contain approved aggregates only and exclude raw rows, payer-card keys, credentials, session tokens, and cross-tenant details.
|
||||
|
||||
@@ -43,7 +43,7 @@ Redis is a private disposable cache and coordination system, not a source of tru
|
||||
- The deployment includes private Redis and on-demand analytics-worker services.
|
||||
- Rate limits and concurrency values remain environment-configurable but changes require security/load evidence and a reviewed spec/config change.
|
||||
- The pre-1.0 Asynq pin creates upgrade risk; CI uses the committed Go module/checksum files and upgrades are deliberate.
|
||||
- Trigger/revision behavior, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests.
|
||||
- Snapshot selection, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
Reference in New Issue
Block a user