Typography note: this standalone backend document uses a system fallback to remain self-contained. The product UI uses the licensed Git-tracked Iran Yekan X FaNum variable webfont. Diagram palette follows the approved ZarinPal-inspired light/dark semantic tokens.
diff --git a/docs/PHYSICAL-SCHEMA-PROPOSAL.md b/docs/PHYSICAL-SCHEMA-PROPOSAL.md index 481dbdb..599a5c3 100644 --- a/docs/PHYSICAL-SCHEMA-PROPOSAL.md +++ b/docs/PHYSICAL-SCHEMA-PROPOSAL.md @@ -1,7 +1,7 @@ # PostgreSQL Physical Schema Specification -- Status: Decision-ready draft — stakeholder approval required -- Scope: Data foundation, demo identity persistence, and analytics invalidation state +- Status: Approved specification — implementation not yet authorized +- Scope: Data foundation, demo identity persistence, and immutable dataset cache identity - Implementation authorization: Not granted - Last updated: 2026-08-20 - Visual companion: [`PHYSICAL-SCHEMA-ERD.html`](./PHYSICAL-SCHEMA-ERD.html) @@ -26,7 +26,7 @@ Define the implementation-ready PostgreSQL shape for the normalized ZarinPal arc 2. A terminal belongs to exactly one merchant. 3. A payment session belongs to exactly one merchant and one terminal; a composite foreign key proves that the terminal belongs to that merchant. 4. A payment attempt belongs to exactly one session and has a positive sequence unique within that session. -5. A bootstrap session belongs to exactly one completed import lineage; a live session has no synthetic import run. +5. Every MVP payment session belongs to exactly one completed import lineage. 6. Attempt provenance is inherited through its session rather than duplicated on every attempt. 7. PSP and issuer-bank codes are reference entities. Switch response codes remain nullable PSP-scoped text because no authoritative meaning catalogue was supplied. 8. Payer-card tokens are nullable, merchant-scoped through the session relationship, and never globally unique. @@ -56,7 +56,7 @@ Define the implementation-ready PostgreSQL shape for the normalized ZarinPal arc | `started_at` | `timestamptz` required | run start | | `completed_at` | `timestamptz` nullable | terminal completion/failure time | -A partial unique index on `(source_sha256, source_schema_version, importer_revision)` where `status = 'completed'` makes a completed import idempotent while permitting explicit retry after a failed run. Terminal state requires `completed_at`; non-terminal state forbids it. +A partial unique index on `(source_sha256, source_schema_version)` where `status = 'completed'` makes a completed source snapshot idempotent across importer rebuilds while permitting explicit retry after a failed run. `importer_revision` remains provenance, not business identity. Terminal state requires `completed_at`; non-terminal state forbids it. #### `import_issues` @@ -206,10 +206,9 @@ The ownership FK guarantees that the user belongs to the recorded merchant. Chec | `session_status` | `text` required | non-empty source lifecycle code | | `created_at` | `timestamptz` required | source Tehran-local value normalized to UTC | | `expires_at` | `timestamptz` required | source `expire_in` normalized to UTC | -| `source_origin` | `text` required | `bootstrap` or `zarinpal_live` | -| `import_run_id` | `bigint` nullable | restrictive FK to `import_runs` | +| `import_run_id` | `bigint` required | restrictive FK to the completed `import_runs` lineage | -The origin check requires an import run for `bootstrap` and forbids one for `zarinpal_live`. Timestamp ordering is not enforced because source ordering anomalies are preserved as evidence rather than rewritten; expiry-before-creation, if found during exact validation, is reported according to the import specification. +Timestamp ordering is not enforced because source ordering anomalies are preserved as evidence rather than rewritten; expiry-before-creation, if found during exact validation, is reported according to the import specification. Approved indexes: @@ -238,27 +237,11 @@ Approved indexes: Unique `(payment_session_id, try_seq)` is the attempt identity and lookup index. A check requires `issuer_bank_id` and `payer_card_key` to be either both null or both populated, matching the authoritative data guide and observed missingness. No timestamp-order check is used because documented anomalies are retained. No standalone payer-token, bank, PSP, or status index is added until an approved measured query justifies it. -### Analytics invalidation +### Static dataset cache namespace -#### `merchant_analytics_revisions` +The MVP performs no normal payment-fact writes after the one-time bootstrap, so the physical schema contains no analytics-revision tables and no insert/update/delete invalidation triggers. The importer permits one completed dataset in the MVP; Redis keys include that completed `import_runs.id` plus its immutable source checksum/schema version, merchant, period, section, and metric-definition version. -`merchant_id bigint` PK/FK to `merchants`; `source_revision bigint` required default `0` and non-negative; `updated_at timestamptz` required. - -#### `cohort_analytics_revisions` - -`category_id bigint` PK/FK to `merchant_categories`; `source_revision bigint` required default `0` and non-negative; `updated_at timestamptz` required. - -Revision values are cache namespace markers, not transaction counts. They may advance more than once in one business operation without changing correctness. - -## Database-managed revision contract - -Direct ZarinPal inserts may bypass Go, so invalidation cannot rely only on application callbacks. Versioned PostgreSQL statement-level `AFTER INSERT` triggers with transition tables update each affected merchant and category once per insert statement: - -- session insert: use merchant/category relationships from the inserted sessions; -- attempt insert: join inserted attempts to their sessions, merchants, and categories; -- update/delete triggers do not exist in MVP because payment facts are append-only. - -This is deliberate schema-owned logic, named in migrations and covered by integration tests. Bulk promotion remains efficient because a multi-row `INSERT ... SELECT` advances each affected revision once, not once per imported row. Trigger failure aborts the fact insert; analytics can never observe a new fact under an old revision. +Dataset replacement and future ZarinPal live writes are outside the MVP. Either change requires a separately approved recovery, namespace-selection, and invalidation design before payment-fact mutation is allowed. ## Staging boundary @@ -266,7 +249,7 @@ Importer staging tables are run-scoped, migration-owned technical structures rat ## Deletion, mutation, and retention -- Imported and live payment facts are append-only in MVP. Corrections require a new specification/ADR and matching invalidation behavior. +- Imported payment facts are immutable in MVP. Corrections or live writes require a new specification/ADR and matching invalidation behavior. - Merchant/category/terminal/import provenance referenced by facts cannot be deleted. Synthetic display names and password hashes may be updated through approved identity workflows. - Session rows are revoked/expired, not hard-deleted during their security lifetime. Authentication audit events expire after 30 days; analytics-access and CSV-export audit events expire after 90 days. Cleanup is allowed only after `expires_at` and must not remove payment/import facts. - Redis analytics entries are disposable and expire after 30 days; no calculated analytics table exists in PostgreSQL. @@ -275,7 +258,7 @@ Importer staging tables are run-scoped, migration-owned technical structures rat - Atlas-generated versioned SQL is reviewed and committed; production startup never auto-migrates. - Migration tests create the schema from zero and upgrade from every supported release state. -- Constraint tests cover terminal ownership, user/audit tenant ownership, no-attempt exclusion, attempt uniqueness, bootstrap/live provenance, paired card/bank nullability, session token digest length, audit fingerprint length, and revision-trigger atomicity. +- Constraint tests cover terminal ownership, user/audit tenant ownership, no-attempt exclusion, attempt uniqueness, bootstrap provenance, paired card/bank nullability, session token digest length, and audit fingerprint length. - Query-plan verification uses production-shaped volumes and `EXPLAIN (ANALYZE, BUFFERS)` for approved analytics, detail pagination, CSV export, and session-authentication paths before adding further indexes. - Initial fact tables are not partitioned. Partitioning requires measured operational need and a separate ADR. @@ -287,20 +270,20 @@ Importer staging tables are run-scoped, migration-owned technical structures rat - Session-level amount, fee, merchant, terminal, status, creation, expiry, origin, and provenance are stored once. - `verify_type` is stored once per terminal and reconciles to every source record for that terminal. - Payer tokens remain unchanged and restricted; they are absent from uniqueness constraints, logs, exports, peer results, and AI inputs. -- Direct session/attempt inserts atomically select new merchant/cohort revision namespaces. +- The sole completed import run provides an immutable dataset cache namespace without database triggers. - Approved period, authentication, drill-down, and export access paths have an explicit initial index or a documented measure-first rule. -- Schema creation, FK/check constraints, trigger behavior, 30/90-day retention cleanup, and query plans are integration-tested before implementation approval can be considered complete. +- Schema creation, FK/check constraints, snapshot selection, 30/90-day retention cleanup, and query plans are integration-tested before implementation approval can be considered complete. ## Boundaries -- Always: preserve source keys and nulls, enforce tenant ownership with FKs, store money at session grain, use UTC instants, version migrations, and test revision changes atomically. +- Always: preserve source keys and nulls, enforce tenant ownership with FKs, store money at session grain, use UTC instants, version migrations, and treat imported payment facts as immutable. - Ask first: add a table/column/index, denormalize tenant/category data, introduce partitioning, classify a new lifecycle status, or permit fact mutation/deletion. - Never: create one wide CSV table, create a fake no-attempt row, duplicate session money on attempts, trust file order, store raw session tokens, or hide production migrations in startup. -## Decision batch for approval +## Resolved decisions 1. Store terminal-invariant `verify_type` on `terminals`, not every payment attempt. 2. Preserve future non-empty lifecycle status values as text and classify them through the metric catalogue instead of using rigid PostgreSQL enums. -3. Use explicit statement-level insert triggers for merchant/cohort analytics revisions because direct ZarinPal writes can bypass Go. +3. Omit cache-revision tables and database triggers because the MVP dataset is immutable; use the sole completed import run as the Redis cache namespace. 4. Persist minimal 30-day authentication audit events and 90-day analytics/export audit events because the approved Identity and API specs require queryable operator evidence without raw payloads, cursors, credentials, or sensitive data. 5. Approve the table, constraint, index, and restrictive-deletion contract above as the migration target. diff --git a/docs/decisions/0007-calendar-aware-analytics-projections.md b/docs/decisions/0007-calendar-aware-analytics-projections.md index 39a60bd..b1bc4a2 100644 --- a/docs/decisions/0007-calendar-aware-analytics-projections.md +++ b/docs/decisions/0007-calendar-aware-analytics-projections.md @@ -17,7 +17,7 @@ Add a reporting-date model containing Gregorian and Jalali date attributes, expl Build idempotent, rebuildable analytics-section queries containing additive components such as session count, attempt count, successful count, amount sum, and fee sum as approved by the metric catalogue. Derive week/month/season totals through the reporting-date model and recompute ratios from their numerators and denominators. -Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under source/cohort and metric-definition revisions. +Calculate non-additive metrics such as exact unique payer-token counts directly at their requested grain; never obtain them by summing daily distinct counts. Cache each typed section in Redis under the immutable completed-import snapshot and metric-definition version. Keep normalized fact tables unpartitioned initially. Add targeted indexes from approved API access paths and verified query plans. Introduce PostgreSQL partitioning only through a future ADR supported by measured table growth, query latency, vacuum/maintenance behavior, or retention operations. @@ -26,5 +26,5 @@ Keep normalized fact tables unpartitioned initially. Add targeted indexes from a - The normalized ERD remains semantically correct while dashboards reuse disposable Redis results. - Section calculations support approved period groupings without storing calculated facts as PostgreSQL business truth. - Explicit reporting-period definitions and metric provenance make future AI observations traceable. -- Lazy parallel calculation, Redis revision keys, job deduplication, partial failure, and SSE visibility tests become required behavior. +- Lazy parallel calculation, dataset-versioned Redis keys, job deduplication, partial failure, and SSE visibility tests become required behavior. - Schema/index/cache details remain subject to data profiling and approved metric/API requirements. diff --git a/docs/decisions/0011-lazy-redis-analytics-cache.md b/docs/decisions/0011-lazy-redis-analytics-cache.md index 653ad9a..8ef3ac0 100644 --- a/docs/decisions/0011-lazy-redis-analytics-cache.md +++ b/docs/decisions/0011-lazy-redis-analytics-cache.md @@ -5,15 +5,15 @@ ## Context -The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Future append-only rows may be written directly by ZarinPal, bypassing Go request handlers. +The hackathon dataset is static after its one-time bootstrap. Continuously refreshing analytics would spend resources before a merchant requests them. A first calculation may contain several independent dashboard sections, and the user should see completed sections without waiting for the slowest query. Production ZarinPal write integration is outside the MVP. ## Decision -On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/source/cohort/metric-definition revision. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled. +On first dashboard access, an authenticated idempotent `POST` creates or returns the current analytics run for that merchant/dataset-snapshot/metric-definition version. It enqueues four section ids—`overview`, `payment_performance`, `fees`, and `peer_comparison`—for bounded parallel calculation by a separate Go worker service. This worker is event-driven, not scheduled. Use `github.com/hibiken/asynq` pinned to `v0.26.0` as the Redis-backed queue/worker library. Asynq is explicitly pre-`v1.0.0`, and its public API may change without a major-version increment. Do not float the dependency or upgrade it without reviewing release notes, API compatibility, Redis compatibility, retry/deduplication behavior, and integration tests. -Cache each completed typed section in Redis using a key derived from the merchant id, merchant source revision, relevant cohort revision, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. Old-revision keys expire naturally and are never treated as current. +Cache each completed typed section in Redis using a key derived from the sole completed import-run snapshot, merchant id, period, metric-definition version, and section id. Use a 30-day TTL. Redis-backed idempotency/single-flight coordination prevents duplicate equivalent jobs. A future explicitly replaced dataset would require a newly specified namespace-selection rule. Expose a separate authenticated `GET` SSE endpoint for the run. Stream stable event ids and typed events for section started, section ready, section failed, overall complete, and heartbeat. Support safe reconnect without exposing another merchant’s jobs or values. One section failure does not suppress sections that succeeded. @@ -27,11 +27,11 @@ Apply authenticated rate limits independently from job deduplication: - manual retry: 3 requests/10 minutes in independent per-user and per-merchant buckets, burst 1; - SSE: 2 concurrent connections per user and 4 per merchant. -Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current revisions and is idempotent; it cannot restart ready siblings. +Return `429 Too Many Requests` with `Retry-After` before enqueuing work. A manual retry targets only one exhausted failed section for the current dataset snapshot and is idempotent; it cannot restart ready siblings. Send SSE heartbeat events every 15 seconds. Retain run events for one hour for authorized `Last-Event-ID` replay. After replay retention expires, return/provide the current run snapshot and continue from new events rather than replaying another merchant’s or an unbounded event history. -Schema-owned PostgreSQL logic atomically increments the affected merchant revision whenever a related payment fact is inserted. Inserts affecting peer benchmarks also increment the corresponding category/cohort revision. This covers direct ZarinPal writes. Payment facts are append-only for the MVP; invalidation for update/delete remains out of scope. +Payment facts are immutable during the MVP, so PostgreSQL contains no cache-revision tables or invalidation triggers. Explicit dataset replacement is an administrative recovery workflow: it selects a different completed import run as the active snapshot and therefore creates a different Redis namespace. Any future live-write integration must define insert/update/delete invalidation in a separately approved specification. Redis is a private disposable cache and coordination system, not a source of truth. PostgreSQL remains authoritative. Cached values contain approved aggregates only and exclude raw rows, payer-card keys, credentials, session tokens, and cross-tenant details. @@ -43,7 +43,7 @@ Redis is a private disposable cache and coordination system, not a source of tru - The deployment includes private Redis and on-demand analytics-worker services. - Rate limits and concurrency values remain environment-configurable but changes require security/load evidence and a reviewed spec/config change. - The pre-1.0 Asynq pin creates upgrade risk; CI uses the committed Go module/checksum files and upgrades are deliberate. -- Trigger/revision behavior, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests. +- Snapshot selection, tenant isolation, job deduplication, partial failures, SSE reconnect, and cache data minimization require integration/security tests. ## References diff --git a/docs/openapi.yaml b/docs/openapi.yaml index cbd7771..31fa096 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -178,7 +178,7 @@ paths: operationId: createAnalyticsRun summary: Create or return the current analytics run description: | - Idempotently resolves the current merchant/source/cohort/metric revision + Idempotently resolves the current merchant/dataset/metric version and starts only missing sections. Equivalent concurrent starts deduplicate. parameters: - $ref: '#/components/parameters/IdempotencyKeyHeader' @@ -1052,8 +1052,7 @@ components: - runId - state - period - - merchantSourceRevision - - cohortRevision + - datasetSnapshotId - metricDefinitionVersion - createdAt - updatedAt @@ -1066,14 +1065,10 @@ components: $ref: '#/components/schemas/AnalyticsRunState' period: $ref: '#/components/schemas/ReportingPeriod' - merchantSourceRevision: + datasetSnapshotId: type: string - pattern: '^[0-9]+$' - description: Decimal string to avoid client integer-width loss. - cohortRevision: - type: string - pattern: '^[0-9]+$' - description: Decimal string to avoid client integer-width loss. + pattern: '^ds_[A-Za-z0-9_-]{8,60}$' + description: Opaque identifier for the immutable completed-import snapshot. metricDefinitionVersion: type: string const: 0.4.0 @@ -1147,14 +1142,14 @@ components: ReadySectionMetadata: type: object additionalProperties: false - required: [calculatedAt, sourceRevision, metricDefinitionVersion] + required: [calculatedAt, datasetSnapshotId, metricDefinitionVersion] properties: calculatedAt: type: string format: date-time - sourceRevision: + datasetSnapshotId: type: string - pattern: '^[0-9]+$' + pattern: '^ds_[A-Za-z0-9_-]{8,60}$' metricDefinitionVersion: type: string const: 0.4.0