docs(cache): remove static dataset invalidation

This commit is contained in:
2026-08-20 18:40:28 +03:30
parent f471eaf3a2
commit 27903ff67e
9 changed files with 62 additions and 102 deletions

View File

@@ -16,7 +16,7 @@ Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive
- Import runs, checksums, schema version, row counts, rejection/error reporting
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
- Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
- PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
- Immutable completed-import snapshot identity for versioned Redis analytics cache keys
- Synthetic merchant display profiles and one demo account per merchant
- Idempotent re-runs and explicit import lifecycle
- Database constraints/indexes driven by documented access paths
@@ -42,7 +42,6 @@ erDiagram
TERMINAL ||--o{ PAYMENT_SESSION : handles
PAYMENT_SESSION ||--o{ PAYMENT_ATTEMPT : contains
IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
MERCHANT_CATEGORY {
@@ -61,6 +60,7 @@ erDiagram
bigint id PK
text source_terminal_key UK
bigint merchant_id FK
text verification_type
}
MERCHANT_USER {
bigint id PK
@@ -90,13 +90,11 @@ erDiagram
text psp_code
text issuer_bank_code
text payer_card_key
text verify_type
int init_time_ms
int verify_time_ms
timestamptz try_created_at
timestamptz verified_at
timestamptz settled_at
timestamptz expires_at
}
IMPORT_RUN {
bigint id PK
@@ -142,27 +140,25 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
A source `(try_seq = 0, try_status = 'NoAttempt')` record creates the session but no `payment_attempt` row; all normalized attempts require `try_seq >= 1`.
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.
8. Populate and validate the approved reporting-date range. The sole completed import run identifies the immutable MVP dataset snapshot. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
9. Never schedule, watch, poll for, or automatically re-import another CSV. Replacing the dataset is an explicit recovery operation outside ordinary service startup.
## Production write and lazy analytics-cache contract
## Static MVP data and lazy analytics-cache contract
- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
- The hackathon MVP reads only the immutable challenge dataset after its one-time PostgreSQL bootstrap.
- No service inserts, updates, or deletes payment facts during normal MVP operation. Future ZarinPal production writes require a separate integration and cache-invalidation specification after acceptance.
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/dataset snapshot. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
- Use Asynq pinned to `v0.26.0` as the Redis-backed task queue. Asynq is pre-`v1.0.0`; never use a floating version, and review API/behavior compatibility before upgrading the pin.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL.
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing the sole completed import-run snapshot, merchant id, selected period, metric-definition version, and section id. Cache entries have a 30-day TTL.
- A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
- Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
- Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings.
- Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
- Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
- No database invalidation trigger or revision table exists for the static MVP. Any future dataset replacement requires a separately approved recovery and cache-namespace-selection contract.
- Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available.
- Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return `429` with `Retry-After` and do not enqueue work.
- SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream.
- Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections.
- Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Manual retry is available only for an exhausted failed section, is idempotent for the current dataset snapshot, and never clears or restarts successful sibling sections.
- Redis is cache/coordination only. PostgreSQL normalized facts and the sole completed import run remain authoritative; loss or flush of Redis causes safe lazy recomputation.
- Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.
## Synthetic merchant/account rules
@@ -189,7 +185,7 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
- Source `adjusted_fee` is normalized once per session as `fee_amount_rial`; retry attempts never multiply fee totals.
- First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready.
- A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation.
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
- Equivalent requests for the immutable completed import snapshot reuse or rebuild the same versioned cache namespace without database triggers.
- Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
- One failed section does not hide ready sections and does not create a ready cache entry.
- A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately.
@@ -218,7 +214,7 @@ Resolved: peer benchmarks use other merchants in the active merchant's category
Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011.
Resolved: the MVP reads one immutable completed-import snapshot and performs no payment-fact mutations. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. No PostgreSQL cache-invalidation trigger exists in the MVP. See ADR 0011.
Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.