zarinpal-analytics-backend/docs/PHYSICAL-SCHEMA-PROPOSAL.md

# PostgreSQL Physical Schema Specification

- Status: Decision-ready draft — stakeholder approval required
- Scope: Data foundation, demo identity persistence, and analytics invalidation state
- Implementation authorization: Not granted
- Last updated: 2026-08-20
- Visual companion: [`PHYSICAL-SCHEMA-ERD.html`](./PHYSICAL-SCHEMA-ERD.html)

## Objective

Define the implementation-ready PostgreSQL shape for the normalized ZarinPal archive without writing migrations. The schema preserves source identity and import provenance, separates the 2,062,839 payment sessions from 1,949,353 real attempts, enforces tenant relationships with foreign keys, supports approved analytics and CSV-detail access paths, and leaves Redis as a disposable result cache.

## Evidence and conventions

- The archive contains 2,213,289 source records, including 263,936 `NoAttempt` markers. A marker creates a session and no attempt row.
- Internal relational keys are `bigint GENERATED ALWAYS AS IDENTITY`. Source identifiers remain non-empty `text` with unique constraints because they are opaque even when they look numeric.
- Canonical instants are `timestamptz` in UTC. A `calendar_dates` dimension provides Tehran-local Gregorian/Jalali reporting attributes.
- Money is `bigint` Rial. `amount_rial` must be positive; `fee_amount_rial` may be zero in production and must not be multiplied by attempt count.
- Flexible upstream lifecycle/status codes remain non-empty `text`, not PostgreSQL enums. Analytics uses a versioned allowlist so a future well-formed status can be preserved and reported as unclassified instead of making direct upstream writes fail.
- All foreign keys are restrictive unless a table definition explicitly says otherwise. Payment and import facts are not cascade-deleted.
- Ent owns application entities and relationships; Atlas produces reviewed versioned SQL; pgx `COPY` is restricted to staging/import paths. This document defines the target contract, not migration SQL.

## Relationship invariants

1. A merchant belongs to exactly one category.
2. A terminal belongs to exactly one merchant.
3. A payment session belongs to exactly one merchant and one terminal; a composite foreign key proves that the terminal belongs to that merchant.
4. A payment attempt belongs to exactly one session and has a positive sequence unique within that session.
5. A bootstrap session belongs to exactly one completed import lineage; a live session has no synthetic import run.
6. Attempt provenance is inherited through its session rather than duplicated on every attempt.
7. PSP and issuer-bank codes are reference entities. Switch response codes remain nullable PSP-scoped text because no authoritative meaning catalogue was supplied.
8. Payer-card tokens are nullable, merchant-scoped through the session relationship, and never globally unique.
9. Verification method is terminal-level. A full archive scan found zero terminals with more than one `verify_type`, while one merchant used different methods across its terminals.

## Physical tables

### Import provenance

#### `import_runs`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `source_sha256` | `text` required | lowercase 64-character SHA-256 hex |
| `source_file_name` | `text` required | basename only; never an operator filesystem path |
| `source_schema_version` | `text` required | explicit supported input contract |
| `importer_revision` | `text` required | immutable code/build revision |
| `status` | `text` required | `created`, `staging`, `validating`, `promoting`, `completed`, or `failed` |
| `observed_rows` | `bigint` required default `0` | source records read, non-negative |
| `accepted_source_rows` | `bigint` required default `0` | accepted source records, non-negative |
| `rejected_source_rows` | `bigint` required default `0` | rejected source records, non-negative |
| `created_sessions` | `bigint` required default `0` | promoted session rows, non-negative |
| `created_attempts` | `bigint` required default `0` | promoted real-attempt rows, non-negative |
| `failure_code` | `text` nullable | stable safe failure category |
| `failure_summary` | `text` nullable | bounded operator-safe summary without raw row data |
| `started_at` | `timestamptz` required | run start |
| `completed_at` | `timestamptz` nullable | terminal completion/failure time |

A partial unique index on `(source_sha256, source_schema_version, importer_revision)` where `status = 'completed'` makes a completed import idempotent while permitting explicit retry after a failed run. Terminal state requires `completed_at`; non-terminal state forbids it.

#### `import_issues`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `import_run_id` | `bigint` required | restrictive FK to `import_runs` |
| `severity` | `text` required | `warning` or `fatal` |
| `rule_code` | `text` required | stable validation rule identifier |
| `affected_rows` | `bigint` required | positive aggregate count |
| `evidence` | `jsonb` required default `{}` | bounded aggregate examples; never payer tokens or credentials |
| `created_at` | `timestamptz` required | evidence time |

Unique `(import_run_id, severity, rule_code)` keeps one aggregate record per rule. Index `(import_run_id, severity)` supports reports.

### Reference data

#### `merchant_categories`

`id bigint` identity PK; `source_category_key text` unique/non-empty; `title_fa text` non-empty; `created_at timestamptz` required.

#### `psps`

`id bigint` identity PK; `source_psp_code text` unique/non-empty; `created_at timestamptz` required. Codes are opaque and do not imply a public PSP identity.

#### `issuer_banks`

`id bigint` identity PK; `source_bank_code text` unique/non-empty; `created_at timestamptz` required. Codes are opaque and do not imply a public bank identity.

#### `calendar_dates`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `gregorian_date` | `date` required | primary key |
| `gregorian_year/month/day` | `smallint` required | validated Gregorian components |
| `jalali_year/month/day` | `smallint` required | month `1..12`, day valid for generated date |
| `iran_weekday` | `smallint` required | `0..6`, Saturday through Friday |
| `reporting_week_start` | `date` required | Saturday containing this date |
| `reporting_week_end_exclusive` | `date` required | following Saturday; equals start + 7 days |
| `persian_season` | `smallint` required | `1..4` |

Unique `(jalali_year, jalali_month, jalali_day)` prevents duplicate calendar mappings. Indexes on `(jalali_year, jalali_month, gregorian_date)`, `(jalali_year, persian_season, gregorian_date)`, and `(reporting_week_start, gregorian_date)` support approved periods. Holiday/named-event columns do not exist.

### Merchant and demo identity

#### `merchants`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `source_merchant_key` | `text` required | unique/non-empty opaque source key |
| `category_id` | `bigint` required | restrictive FK to `merchant_categories` |
| `display_name` | `text` required | generated demo-facing name |
| `profile_is_synthetic` | `boolean` required | true for generated profile data |
| `created_at` | `timestamptz` required | creation time |

Index `category_id` supports peer cohort membership.

#### `terminals`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `source_terminal_key` | `text` required | unique/non-empty opaque source key |
| `merchant_id` | `bigint` required | restrictive FK to `merchants` |
| `verification_type` | `text` required | `Automated` or `Manual` for the approved source contract |
| `created_at` | `timestamptz` required | creation time |

Unique `(id, merchant_id)` is the target of the session ownership FK. Index `merchant_id` supports merchant terminal lookup.

#### `merchant_users`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_id` | `bigint` required | restrictive FK to `merchants` |
| `username_normalized` | `text` required | lowercase ASCII, globally unique |
| `password_hash` | `text` required | complete Argon2id encoded hash only |
| `is_demo` | `boolean` required default `true` | demo adapter marker |
| `disabled_at` | `timestamptz` nullable | disabled accounts cannot authenticate |
| `created_at/updated_at` | `timestamptz` required | lifecycle timestamps |

Unique `(id, merchant_id)` is the target of tenant-ownership audit FKs. A partial unique index on `merchant_id WHERE is_demo` enforces at most one seeded demo user per merchant without preventing a future non-demo identity migration.

#### `auth_sessions`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_user_id` | `bigint` required | restrictive FK to `merchant_users` |
| `token_digest` | `bytea` required | unique 32-byte SHA-256 digest; never raw token |
| `created_at` | `timestamptz` required | creation time |
| `last_activity_at` | `timestamptz` required | coalesced activity time |
| `idle_expires_at` | `timestamptz` required | no more than 12 hours after approved activity |
| `absolute_expires_at` | `timestamptz` required | no more than 7 days after creation |
| `revoked_at` | `timestamptz` nullable | immediate invalidation time |
| `revocation_reason` | `text` nullable | safe bounded reason code |

Checks enforce `last_activity_at >= created_at`, `idle_expires_at <= absolute_expires_at`, and paired revocation fields. Unique `token_digest` is the lookup path. Index `(merchant_user_id, revoked_at, absolute_expires_at, created_at)` supports the two-active-session rule and revocation.

#### `auth_audit_events`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_user_id` | `bigint` nullable | FK to `merchant_users`; null for an unresolved login subject |
| `event_type` | `text` required | stable approved security-event code |
| `outcome` | `text` required | safe outcome category |
| `correlation_id` | `text` nullable | request correlation only, never token/credential material |
| `occurred_at` | `timestamptz` required | event time |
| `expires_at` | `timestamptz` required | 30-day retention deadline |

Index `expires_at` supports bounded cleanup; `(merchant_user_id, occurred_at DESC)` supports tenant-safe investigation. Raw IP addresses, user agents, usernames attempted, passwords, hashes, and token values are prohibited.

#### `analytics_access_audit_events`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `merchant_user_id` | `bigint` required | restrictive FK to `merchant_users` |
| `merchant_id` | `bigint` required | part of composite ownership FK with `merchant_user_id` |
| `event_type` | `text` required | approved analytics/detail/export action code |
| `result` | `text` required | stable completed/rejected/aborted outcome code |
| `period_id` | `text` nullable | approved canonical period identifier |
| `view_type` | `text` nullable | approved detail/export view code |
| `filter_fingerprint` | `bytea` nullable | 32-byte digest of canonical filters; never raw filter/cursor data |
| `row_count` | `bigint` nullable | non-negative completed export count only |
| `byte_count` | `bigint` nullable | non-negative completed export byte count only |
| `request_id` | `text` nullable | bounded safe correlation identifier |
| `occurred_at` | `timestamptz` required | action time |
| `expires_at` | `timestamptz` required | 90-day retention deadline |

The ownership FK guarantees that the user belongs to the recorded merchant. Checks enforce a 32-byte fingerprint when present and non-negative row/byte counts. Index `expires_at` supports automatic retention cleanup; `(merchant_id, occurred_at DESC)` and `(merchant_user_id, occurred_at DESC)` support operator-only investigation. Exported values, payer tokens, raw cursors/filters, credentials, user-controlled filenames, and peer data are prohibited. This table has no merchant-facing API or UI.

### Payment facts

#### `payment_sessions`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `source_session_key` | `text` required | globally unique/non-empty opaque source key |
| `merchant_id` | `bigint` required | restrictive FK to `merchants` |
| `terminal_id` | `bigint` required | part of composite FK `(terminal_id, merchant_id)` to `terminals` |
| `amount_rial` | `bigint` required | positive session amount |
| `fee_amount_rial` | `bigint` required | non-negative real merchant fee, stored once |
| `session_status` | `text` required | non-empty source lifecycle code |
| `created_at` | `timestamptz` required | source Tehran-local value normalized to UTC |
| `expires_at` | `timestamptz` required | source `expire_in` normalized to UTC |
| `source_origin` | `text` required | `bootstrap` or `zarinpal_live` |
| `import_run_id` | `bigint` nullable | restrictive FK to `import_runs` |

The origin check requires an import run for `bootstrap` and forbids one for `zarinpal_live`. Timestamp ordering is not enforced because source ordering anomalies are preserved as evidence rather than rewritten; expiry-before-creation, if found during exact validation, is reported according to the import specification.

Approved indexes:

- unique `source_session_key`
- `(merchant_id, created_at DESC, id DESC)` for period lists and keyset export
- `(terminal_id, created_at DESC, id DESC)` for terminal drill-down
- `(merchant_id, session_status, created_at DESC, id DESC)` for paid/unverified/failed detail views

#### `payment_attempts`

| Column | Type/nullability | Constraint or meaning |
|---|---|---|
| `id` | `bigint` required | identity primary key |
| `payment_session_id` | `bigint` required | restrictive FK to `payment_sessions` |
| `try_seq` | `smallint` required | positive source sequence |
| `try_status` | `text` required | non-empty source lifecycle code; never `NoAttempt` |
| `psp_id` | `bigint` required | restrictive FK to `psps` |
| `switch_response_code` | `text` nullable | PSP-scoped raw code |
| `issuer_bank_id` | `bigint` nullable | restrictive FK to `issuer_banks` |
| `payer_card_key` | `text` nullable | unchanged restricted opaque token |
| `init_time_ms` | `integer` nullable | non-negative API duration |
| `verify_time_ms` | `integer` nullable | non-negative API duration |
| `try_created_at` | `timestamptz` required | source attempt creation normalized to UTC |
| `verified_at` | `timestamptz` nullable | preserved source instant |
| `settled_at` | `timestamptz` nullable | preserved source instant |

Unique `(payment_session_id, try_seq)` is the attempt identity and lookup index. A check requires `issuer_bank_id` and `payer_card_key` to be either both null or both populated, matching the authoritative data guide and observed missingness. No timestamp-order check is used because documented anomalies are retained. No standalone payer-token, bank, PSP, or status index is added until an approved measured query justifies it.

### Analytics invalidation

#### `merchant_analytics_revisions`

`merchant_id bigint` PK/FK to `merchants`; `source_revision bigint` required default `0` and non-negative; `updated_at timestamptz` required.

#### `cohort_analytics_revisions`

`category_id bigint` PK/FK to `merchant_categories`; `source_revision bigint` required default `0` and non-negative; `updated_at timestamptz` required.

Revision values are cache namespace markers, not transaction counts. They may advance more than once in one business operation without changing correctness.

## Database-managed revision contract

Direct ZarinPal inserts may bypass Go, so invalidation cannot rely only on application callbacks. Versioned PostgreSQL statement-level `AFTER INSERT` triggers with transition tables update each affected merchant and category once per insert statement:

- session insert: use merchant/category relationships from the inserted sessions;
- attempt insert: join inserted attempts to their sessions, merchants, and categories;
- update/delete triggers do not exist in MVP because payment facts are append-only.

This is deliberate schema-owned logic, named in migrations and covered by integration tests. Bulk promotion remains efficient because a multi-row `INSERT ... SELECT` advances each affected revision once, not once per imported row. Trigger failure aborts the fact insert; analytics can never observe a new fact under an old revision.

## Staging boundary

Importer staging tables are run-scoped, migration-owned technical structures rather than durable domain entities. They mirror raw columns plus `import_run_id`, `source_row_number`, and parse/validation metadata. They have no application Ent entity, no long-term API, and are truncated/dropped only by the explicit recovery lifecycle after the durable import report is complete. Promotion targets only the tables above.

## Deletion, mutation, and retention

- Imported and live payment facts are append-only in MVP. Corrections require a new specification/ADR and matching invalidation behavior.
- Merchant/category/terminal/import provenance referenced by facts cannot be deleted. Synthetic display names and password hashes may be updated through approved identity workflows.
- Session rows are revoked/expired, not hard-deleted during their security lifetime. Authentication audit events expire after 30 days; analytics-access and CSV-export audit events expire after 90 days. Cleanup is allowed only after `expires_at` and must not remove payment/import facts.
- Redis analytics entries are disposable and expire after 30 days; no calculated analytics table exists in PostgreSQL.

## Migration and verification requirements

- Atlas-generated versioned SQL is reviewed and committed; production startup never auto-migrates.
- Migration tests create the schema from zero and upgrade from every supported release state.
- Constraint tests cover terminal ownership, user/audit tenant ownership, no-attempt exclusion, attempt uniqueness, bootstrap/live provenance, paired card/bank nullability, session token digest length, audit fingerprint length, and revision-trigger atomicity.
- Query-plan verification uses production-shaped volumes and `EXPLAIN (ANALYZE, BUFFERS)` for approved analytics, detail pagination, CSV export, and session-authentication paths before adding further indexes.
- Initial fact tables are not partitioned. Partitioning requires measured operational need and a separate ADR.

## Acceptance criteria

- The physical schema represents all 2,062,839 sessions and exactly 1,949,353 real attempts without a fake `NoAttempt` attempt.
- Every session's terminal is provably owned by the same merchant through a database foreign key.
- Every source session key and `(session, try_seq)` pair is unique after promotion.
- Session-level amount, fee, merchant, terminal, status, creation, expiry, origin, and provenance are stored once.
- `verify_type` is stored once per terminal and reconciles to every source record for that terminal.
- Payer tokens remain unchanged and restricted; they are absent from uniqueness constraints, logs, exports, peer results, and AI inputs.
- Direct session/attempt inserts atomically select new merchant/cohort revision namespaces.
- Approved period, authentication, drill-down, and export access paths have an explicit initial index or a documented measure-first rule.
- Schema creation, FK/check constraints, trigger behavior, 30/90-day retention cleanup, and query plans are integration-tested before implementation approval can be considered complete.

## Boundaries

- Always: preserve source keys and nulls, enforce tenant ownership with FKs, store money at session grain, use UTC instants, version migrations, and test revision changes atomically.
- Ask first: add a table/column/index, denormalize tenant/category data, introduce partitioning, classify a new lifecycle status, or permit fact mutation/deletion.
- Never: create one wide CSV table, create a fake no-attempt row, duplicate session money on attempts, trust file order, store raw session tokens, or hide production migrations in startup.

## Decision batch for approval

1. Store terminal-invariant `verify_type` on `terminals`, not every payment attempt.
2. Preserve future non-empty lifecycle status values as text and classify them through the metric catalogue instead of using rigid PostgreSQL enums.
3. Use explicit statement-level insert triggers for merchant/cohort analytics revisions because direct ZarinPal writes can bypass Go.
4. Persist minimal 30-day authentication audit events and 90-day analytics/export audit events because the approved Identity and API specs require queryable operator evidence without raw payloads, cursors, credentials, or sensitive data.
5. Approve the table, constraint, index, and restrictive-deletion contract above as the migration target.