From 96a27e1b98a8f05c1bc2793b8b1f00584001f385 Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 16:41:07 +0330 Subject: [PATCH] docs: pin analytics queue safety limits --- docs/DATA-FOUNDATION-SPEC.md | 11 ++++++++-- .../0011-lazy-redis-analytics-cache.md | 22 ++++++++++++++++++- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/docs/DATA-FOUNDATION-SPEC.md b/docs/DATA-FOUNDATION-SPEC.md index 7e39ffa..0d5c164 100644 --- a/docs/DATA-FOUNDATION-SPEC.md +++ b/docs/DATA-FOUNDATION-SPEC.md @@ -149,12 +149,17 @@ This contract applies only to the one-time challenge-data bootstrap and explicit - This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow. - Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract. - An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. +- Use Asynq pinned to `v0.26.0` as the Redis-backed task queue. Asynq is pre-`v1.0.0`; never use a floating version, and review API/behavior compatibility before upgrading the pin. - Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL. - A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish. - Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant. +- Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings. - Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers. - Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan. - Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available. +- Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return `429` with `Retry-After` and do not enqueue work. +- SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream. +- Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections. - Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation. - Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details. @@ -185,6 +190,8 @@ This contract applies only to the one-time challenge-data bootstrap and explicit - Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out. - One failed section does not hide ready sections and does not create a ready cache entry. - A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately. +- Start, manual-retry, and SSE connection limits return a typed `429` with `Retry-After`; rejected requests enqueue no jobs and cannot affect another merchant. +- SSE emits heartbeats within 15 seconds and supports authorized replay for one hour; replay expiry falls back to the current run snapshot without duplicating work. - Redis loss produces a cold-cache calculation rather than loss of source data. - Reporting-date rows covering the dataset range reconcile deterministically between Gregorian and Jalali day, week, month, and season attributes. - Every reporting week starts Saturday at `00:00:00` and ends Friday at `23:59:59.999…` in `Asia/Tehran`; API ranges use an equivalent half-open `[start, next_start)` interval. @@ -205,7 +212,7 @@ Resolved: peer benchmarks exclude the active merchant and require at least 10 ot Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007. -Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four parallel sections in a separate Go worker; Redis caches each typed result for 30 days; an authenticated `GET` SSE stream reveals sections as ready. Each section has a 60-second attempt timeout and at most two transient retries using exponential backoff/jitter. PostgreSQL merchant/cohort revisions provide logical invalidation. See ADR 0011. +Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011. Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010. @@ -213,4 +220,4 @@ Resolved: reporting weeks follow the Iranian week: Saturday through Friday in `A Resolved: only source session status `Verified` is a definitely successful payment. Keep `Paid` separate until an authoritative ZarinPal definition changes the metric catalogue. See `METRIC-CATALOGUE.md`. -No broad architecture approval gates remain for this capability. Physical columns, indexes, exact SSE schemas/reconnect retention, worker concurrency, retry classification, analytics-section response contracts, and migration details still require acceptance in the implementation-ready specs. +No broad architecture approval gates remain for this capability. Physical columns, exact SSE event schemas, retryable-error classification, analytics-section response contracts, observability targets, and migration details still require acceptance in the implementation-ready specs. diff --git a/docs/decisions/0011-lazy-redis-analytics-cache.md b/docs/decisions/0011-lazy-redis-analytics-cache.md index 49bff88..653ad9a 100644 --- a/docs/decisions/0011-lazy-redis-analytics-cache.md +++ b/docs/decisions/0011-lazy-redis-analytics-cache.md @@ -11,12 +11,26 @@ The hackathon dataset is static after its one-time bootstrap. Continuously refre On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/source/cohort/metric-definition revision. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled. +Use `github.com/hibiken/asynq` pinned to `v0.26.0` as the Redis-backed queue/worker library. Asynq is explicitly pre-`v1.0.0`, and its public API may change without a major-version increment. Do not float the dependency or upgrade it without reviewing release notes, API compatibility, Redis compatibility, retry/deduplication behavior, and integration tests. + Cache each completed typed section in Redis using a key derived from the merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. Old-revision keys expire naturally and are never treated as current. Expose a separate authenticated `GET` SSE endpoint for the run. Stream stable event ids and typed events for section started, section ready, section failed, overall complete, and heartbeat. Support safe reconnect without exposing another merchant’s jobs or values. One section failure does not suppress sections that succeeded. Give each section attempt a 60-second timeout. Retry only classified transient failures at most twice, for a maximum of three attempts. Use exponential backoff with jitter (initial nominal delay 1 second, then 2 seconds; jitter prevents synchronized retry bursts). Do not retry authorization, validation, unsupported-metric, or other deterministic failures. +Default worker concurrency to 16 jobs globally and 4 per merchant. Enforce the global value in the worker and the merchant value in queue middleware/coordination; both are validated environment settings. + +Apply authenticated rate limits independently from job deduplication: + +- analytics start: 10 requests/minute in independent per-user and per-merchant buckets, burst 3; +- manual retry: 3 requests/10 minutes in independent per-user and per-merchant buckets, burst 1; +- SSE: 2 concurrent connections per user and 4 per merchant. + +Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current revisions and is idempotent; it cannot restart ready siblings. + +Send SSE heartbeat events every 15 seconds. Retain run events for one hour for authorized `Last-Event-ID` replay. After replay retention expires, return/provide the current run snapshot and continue from new events rather than replaying another merchant’s or an unbounded event history. + Schema-owned PostgreSQL logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts affecting peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes. Payment facts are append-only for the MVP; invalidation for update/delete remains out of scope. Redis is a private disposable cache and coordination system, not a source of truth. PostgreSQL remains authoritative. Cached values contain approved aggregates only and exclude raw rows, payer-card keys, credentials, session tokens, and cross-tenant details. @@ -27,5 +41,11 @@ Redis is a private disposable cache and coordination system, not a source of tru - Redis loss or key expiry causes safe lazy recomputation; Redis is excluded from backup/restore requirements. - No scheduled refresh or time-based freshness claim is required. - The deployment includes private Redis and on-demand analytics-worker services. -- Global/per-merchant concurrency and SSE connection/replay-retention limits still require explicit values. +- Rate limits and concurrency values remain environment-configurable but changes require security/load evidence and a reviewed spec/config change. +- The pre-1.0 Asynq pin creates upgrade risk; CI uses the committed Go module/checksum files and upgrades are deliberate. - Trigger/revision behavior, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests. + +## References + +- Asynq repository and stability notice: +- Pinned release: