docs(cache): remove static dataset invalidation
This commit is contained in:
@@ -16,7 +16,7 @@ Perform a one-time, repeatable bootstrap of the supplied payment-attempt archive
|
||||
- Import runs, checksums, schema version, row counts, rejection/error reporting
|
||||
- Normalized merchant, category, terminal, payment-session, payment-attempt, and account/profile concepts
|
||||
- Gregorian/Jalali reporting dates with day, week, month, and Persian-season attributes
|
||||
- PostgreSQL merchant/cohort invalidation revisions for versioned Redis analytics cache keys
|
||||
- Immutable completed-import snapshot identity for versioned Redis analytics cache keys
|
||||
- Synthetic merchant display profiles and one demo account per merchant
|
||||
- Idempotent re-runs and explicit import lifecycle
|
||||
- Database constraints/indexes driven by documented access paths
|
||||
@@ -42,7 +42,6 @@ erDiagram
|
||||
TERMINAL ||--o{ PAYMENT_SESSION : handles
|
||||
PAYMENT_SESSION ||--o{ PAYMENT_ATTEMPT : contains
|
||||
IMPORT_RUN ||--o{ PAYMENT_SESSION : introduced_by
|
||||
IMPORT_RUN ||--o{ PAYMENT_ATTEMPT : introduced_by
|
||||
IMPORT_RUN ||--o{ IMPORT_ISSUE : reports
|
||||
|
||||
MERCHANT_CATEGORY {
|
||||
@@ -61,6 +60,7 @@ erDiagram
|
||||
bigint id PK
|
||||
text source_terminal_key UK
|
||||
bigint merchant_id FK
|
||||
text verification_type
|
||||
}
|
||||
MERCHANT_USER {
|
||||
bigint id PK
|
||||
@@ -90,13 +90,11 @@ erDiagram
|
||||
text psp_code
|
||||
text issuer_bank_code
|
||||
text payer_card_key
|
||||
text verify_type
|
||||
int init_time_ms
|
||||
int verify_time_ms
|
||||
timestamptz try_created_at
|
||||
timestamptz verified_at
|
||||
timestamptz settled_at
|
||||
timestamptz expires_at
|
||||
}
|
||||
IMPORT_RUN {
|
||||
bigint id PK
|
||||
@@ -142,27 +140,25 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
|
||||
A source `(try_seq = 0, try_status = 'NoAttempt')` record creates the session but no `payment_attempt` row; all normalized attempts require `try_seq >= 1`.
|
||||
6. Promote accepted data transactionally or through a resumable, explicitly versioned workflow; a failed run cannot appear complete.
|
||||
7. Emit a machine-readable and human-readable import report. Do not log raw card keys or credentials.
|
||||
8. Populate and validate the approved reporting-date range, then initialize merchant/cohort analytics revisions. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
|
||||
9. Never schedule, watch, or poll for another CSV. After bootstrap, production source-row creation is an external ZarinPal responsibility. Insert-triggered revisions make prior Redis cache keys obsolete.
|
||||
8. Populate and validate the approved reporting-date range. The sole completed import run identifies the immutable MVP dataset snapshot. Do not eagerly calculate dashboard results; first relevant access starts lazy section jobs.
|
||||
9. Never schedule, watch, poll for, or automatically re-import another CSV. Replacing the dataset is an explicit recovery operation outside ordinary service startup.
|
||||
|
||||
## Production write and lazy analytics-cache contract
|
||||
## Static MVP data and lazy analytics-cache contract
|
||||
|
||||
- ZarinPal is the authoritative writer of future production payment sessions/attempts in the normalized PostgreSQL schema.
|
||||
- This project owns the normalized schema contract, migrations, read models, and analytics; it does not recreate ZarinPal's payment-write workflow.
|
||||
- Payment facts are append-only in the MVP. Existing session/attempt updates and hard deletion are out of scope; a future mutation requirement needs a new invalidation contract.
|
||||
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/revision. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
|
||||
- The hackathon MVP reads only the immutable challenge dataset after its one-time PostgreSQL bootstrap.
|
||||
- No service inserts, updates, or deletes payment facts during normal MVP operation. Future ZarinPal production writes require a separate integration and cache-invalidation specification after acceptance.
|
||||
- An idempotent authenticated `POST` starts or returns the current analytics run for the merchant/dataset snapshot. It enqueues the four MVP sections—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service.
|
||||
- Use Asynq pinned to `v0.26.0` as the Redis-backed task queue. Asynq is pre-`v1.0.0`; never use a floating version, and review API/behavior compatibility before upgrading the pin.
|
||||
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Cache entries have a 30-day TTL.
|
||||
- Each completed section is serialized through its approved typed contract and cached in Redis under a key containing the sole completed import-run snapshot, merchant id, selected period, metric-definition version, and section id. Cache entries have a 30-day TTL.
|
||||
- A separate authenticated `GET` SSE stream for that run emits section progress and makes each successful section available immediately; the dashboard does not wait for every job to finish.
|
||||
- Redis-backed idempotency/single-flight coordination prevents equivalent concurrent requests from duplicating the same section calculation. Parallelism is bounded globally and per merchant.
|
||||
- Worker concurrency defaults to 16 jobs globally and 4 jobs per merchant, configurable downward/upward through validated environment settings.
|
||||
- Every inserted related payment fact atomically advances the affected PostgreSQL merchant analytics revision. It also advances the relevant cohort/category revision when peer metrics can change. Database-managed revision changes cover direct ZarinPal inserts that bypass Go request handlers.
|
||||
- Revision changes invalidate logically by producing new cache keys; old values are not trusted and expire through TTL without requiring a Redis key scan.
|
||||
- No database invalidation trigger or revision table exists for the static MVP. Any future dataset replacement requires a separately approved recovery and cache-namespace-selection contract.
|
||||
- Each section attempt has a 60-second timeout. Transient failures receive at most two automatic retries (three total attempts) with exponential backoff and jitter; deterministic validation/authorization failures are not retried. Exhausted sections emit a typed failure event without marking the section ready. Other successful sections remain available.
|
||||
- Rate limits use independent authenticated-user and merchant buckets: analytics starts allow 10 requests/minute with burst 3; manual failed-section retries allow 3 requests/10 minutes with burst 1; SSE allows 2 concurrent connections/user and 4/merchant. Rejections return `429` with `Retry-After` and do not enqueue work.
|
||||
- SSE sends a heartbeat every 15 seconds and retains replayable run events for one hour. Reconnects resume from the last authorized event id where available; after retention, the client fetches the current run snapshot and reopens the stream.
|
||||
- Manual retry is available only for an exhausted failed section, is idempotent for the current revisions, and never clears or restarts successful sibling sections.
|
||||
- Redis is cache/coordination only. PostgreSQL normalized facts and revision rows remain authoritative; loss or flush of Redis causes safe lazy recomputation.
|
||||
- Manual retry is available only for an exhausted failed section, is idempotent for the current dataset snapshot, and never clears or restarts successful sibling sections.
|
||||
- Redis is cache/coordination only. PostgreSQL normalized facts and the sole completed import run remain authoritative; loss or flush of Redis causes safe lazy recomputation.
|
||||
- Redis and the on-demand job worker are private services, never exposed through Nginx. Cache values exclude payer-card keys, credentials, raw payment rows, and cross-tenant details.
|
||||
|
||||
## Synthetic merchant/account rules
|
||||
@@ -189,7 +185,7 @@ This contract applies only to the one-time challenge-data bootstrap and explicit
|
||||
- Source `adjusted_fee` is normalized once per session as `fee_amount_rial`; retry attempts never multiply fee totals.
|
||||
- First access idempotently creates/returns one run, starts all four core analytics sections in bounded parallel, and delivers each through authenticated SSE as soon as it is ready.
|
||||
- A second equivalent request reads current 30-day Redis entries without repeating source-fact aggregation.
|
||||
- Inserting a related payment fact advances only the affected merchant revision and any affected cohort revision; the next relevant request recomputes once and publishes atomically.
|
||||
- Equivalent requests for the immutable completed import snapshot reuse or rebuild the same versioned cache namespace without database triggers.
|
||||
- Concurrent equivalent requests do not duplicate section jobs; configured global/per-merchant limits prevent unbounded query fan-out.
|
||||
- One failed section does not hide ready sections and does not create a ready cache entry.
|
||||
- A transient section failure performs no more than two retries with exponential backoff/jitter; each attempt ends within 60 seconds, while non-retryable failures fail immediately.
|
||||
@@ -218,7 +214,7 @@ Resolved: peer benchmarks use other merchants in the active merchant's category
|
||||
|
||||
Resolved: use normalized PostgreSQL source-of-truth entities plus a reporting-date dimension, then compute versioned analytics sections lazily into Redis. Daily, weekly, Jalali-month, and Persian-season results use explicit date boundaries. Keep fact tables unpartitioned initially and introduce partitioning only from measured need. See ADR 0007.
|
||||
|
||||
Resolved: ZarinPal writes future append-only production rows directly to normalized tables. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. PostgreSQL revisions provide invalidation. See ADR 0011.
|
||||
Resolved: the MVP reads one immutable completed-import snapshot and performs no payment-fact mutations. An idempotent `POST` starts four sections through Asynq `v0.26.0` (explicitly pre-1.0) in a separate Go worker, limited to 4 concurrent jobs/merchant and 16 globally. Redis caches results for 30 days; authenticated SSE uses 15-second heartbeats and one-hour replay. Sections have a 60-second attempt timeout and two transient exponential-backoff retries. Start, SSE, and manual-retry operations are rate-limited. No PostgreSQL cache-invalidation trigger exists in the MVP. See ADR 0011.
|
||||
|
||||
Resolved: holiday and named-event reporting are removed from scope. MVP reporting periods are limited to daily, weekly, Jalali-month, and Persian-season views. See ADR 0010.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user