zarinpal-analytics-backend/docs/PHYSICAL-SCHEMA-PROPOSAL.md

PostgreSQL Physical Schema Specification

• Status: Decision-ready draft — stakeholder approval required
• Scope: Data foundation, demo identity persistence, and analytics invalidation state
• Implementation authorization: Not granted
• Last updated: 2026-08-20
• Visual companion: PHYSICAL-SCHEMA-ERD.html

Objective

Define the implementation-ready PostgreSQL shape for the normalized ZarinPal archive without writing migrations. The schema preserves source identity and import provenance, separates the 2,062,839 payment sessions from 1,949,353 real attempts, enforces tenant relationships with foreign keys, supports approved analytics and CSV-detail access paths, and leaves Redis as a disposable result cache.

Evidence and conventions

• The archive contains 2,213,289 source records, including 263,936 NoAttempt markers. A marker creates a session and no attempt row.
• Internal relational keys are bigint GENERATED ALWAYS AS IDENTITY. Source identifiers remain non-empty text with unique constraints because they are opaque even when they look numeric.
• Canonical instants are timestamptz in UTC. A calendar_dates dimension provides Tehran-local Gregorian/Jalali reporting attributes.
• Money is bigint Rial. amount_rial must be positive; fee_amount_rial may be zero in production and must not be multiplied by attempt count.
• Flexible upstream lifecycle/status codes remain non-empty text, not PostgreSQL enums. Analytics uses a versioned allowlist so a future well-formed status can be preserved and reported as unclassified instead of making direct upstream writes fail.
• All foreign keys are restrictive unless a table definition explicitly says otherwise. Payment and import facts are not cascade-deleted.
• Ent owns application entities and relationships; Atlas produces reviewed versioned SQL; pgx COPY is restricted to staging/import paths. This document defines the target contract, not migration SQL.

Relationship invariants

1. A merchant belongs to exactly one category.
2. A terminal belongs to exactly one merchant.
3. A payment session belongs to exactly one merchant and one terminal; a composite foreign key proves that the terminal belongs to that merchant.
4. A payment attempt belongs to exactly one session and has a positive sequence unique within that session.
5. A bootstrap session belongs to exactly one completed import lineage; a live session has no synthetic import run.
6. Attempt provenance is inherited through its session rather than duplicated on every attempt.
7. PSP and issuer-bank codes are reference entities. Switch response codes remain nullable PSP-scoped text because no authoritative meaning catalogue was supplied.
8. Payer-card tokens are nullable, merchant-scoped through the session relationship, and never globally unique.
9. Verification method is terminal-level. A full archive scan found zero terminals with more than one verify_type, while one merchant used different methods across its terminals.

Physical tables

Import provenance

import_runs

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
source_sha256	text required	lowercase 64-character SHA-256 hex
source_file_name	text required	basename only; never an operator filesystem path
source_schema_version	text required	explicit supported input contract
importer_revision	text required	immutable code/build revision
status	text required	created, staging, validating, promoting, completed, or failed
observed_rows	bigint required default 0	source records read, non-negative
accepted_source_rows	bigint required default 0	accepted source records, non-negative
rejected_source_rows	bigint required default 0	rejected source records, non-negative
created_sessions	bigint required default 0	promoted session rows, non-negative
created_attempts	bigint required default 0	promoted real-attempt rows, non-negative
failure_code	text nullable	stable safe failure category
failure_summary	text nullable	bounded operator-safe summary without raw row data
started_at	timestamptz required	run start
completed_at	timestamptz nullable	terminal completion/failure time

A partial unique index on (source_sha256, source_schema_version, importer_revision) where status = 'completed' makes a completed import idempotent while permitting explicit retry after a failed run. Terminal state requires completed_at; non-terminal state forbids it.

import_issues

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
import_run_id	bigint required	restrictive FK to import_runs
severity	text required	warning or fatal
rule_code	text required	stable validation rule identifier
affected_rows	bigint required	positive aggregate count
evidence	jsonb required default {}	bounded aggregate examples; never payer tokens or credentials
created_at	timestamptz required	evidence time

Unique (import_run_id, severity, rule_code) keeps one aggregate record per rule. Index (import_run_id, severity) supports reports.

Reference data

merchant_categories

id bigint identity PK; source_category_key text unique/non-empty; title_fa text non-empty; created_at timestamptz required.

psps

id bigint identity PK; source_psp_code text unique/non-empty; created_at timestamptz required. Codes are opaque and do not imply a public PSP identity.

issuer_banks

id bigint identity PK; source_bank_code text unique/non-empty; created_at timestamptz required. Codes are opaque and do not imply a public bank identity.

calendar_dates

Column	Type/nullability	Constraint or meaning
gregorian_date	date required	primary key
gregorian_year/month/day	smallint required	validated Gregorian components
jalali_year/month/day	smallint required	month 1..12, day valid for generated date
iran_weekday	smallint required	0..6, Saturday through Friday
reporting_week_start	date required	Saturday containing this date
reporting_week_end_exclusive	date required	following Saturday; equals start + 7 days
persian_season	smallint required	1..4

Unique (jalali_year, jalali_month, jalali_day) prevents duplicate calendar mappings. Indexes on (jalali_year, jalali_month, gregorian_date), (jalali_year, persian_season, gregorian_date), and (reporting_week_start, gregorian_date) support approved periods. Holiday/named-event columns do not exist.

Merchant and demo identity

merchants

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
source_merchant_key	text required	unique/non-empty opaque source key
category_id	bigint required	restrictive FK to merchant_categories
display_name	text required	generated demo-facing name
profile_is_synthetic	boolean required	true for generated profile data
created_at	timestamptz required	creation time

Index category_id supports peer cohort membership.

terminals

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
source_terminal_key	text required	unique/non-empty opaque source key
merchant_id	bigint required	restrictive FK to merchants
verification_type	text required	Automated or Manual for the approved source contract
created_at	timestamptz required	creation time

Unique (id, merchant_id) is the target of the session ownership FK. Index merchant_id supports merchant terminal lookup.

merchant_users

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
merchant_id	bigint required	restrictive FK to merchants
username_normalized	text required	lowercase ASCII, globally unique
password_hash	text required	complete Argon2id encoded hash only
is_demo	boolean required default true	demo adapter marker
disabled_at	timestamptz nullable	disabled accounts cannot authenticate
created_at/updated_at	timestamptz required	lifecycle timestamps

Unique (id, merchant_id) is the target of tenant-ownership audit FKs. A partial unique index on merchant_id WHERE is_demo enforces at most one seeded demo user per merchant without preventing a future non-demo identity migration.

auth_sessions

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
merchant_user_id	bigint required	restrictive FK to merchant_users
token_digest	bytea required	unique 32-byte SHA-256 digest; never raw token
created_at	timestamptz required	creation time
last_activity_at	timestamptz required	coalesced activity time
idle_expires_at	timestamptz required	no more than 12 hours after approved activity
absolute_expires_at	timestamptz required	no more than 7 days after creation
revoked_at	timestamptz nullable	immediate invalidation time
revocation_reason	text nullable	safe bounded reason code

Checks enforce last_activity_at >= created_at, idle_expires_at <= absolute_expires_at, and paired revocation fields. Unique token_digest is the lookup path. Index (merchant_user_id, revoked_at, absolute_expires_at, created_at) supports the two-active-session rule and revocation.

auth_audit_events

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
merchant_user_id	bigint nullable	FK to merchant_users; null for an unresolved login subject
event_type	text required	stable approved security-event code
outcome	text required	safe outcome category
correlation_id	text nullable	request correlation only, never token/credential material
occurred_at	timestamptz required	event time
expires_at	timestamptz required	30-day retention deadline

Index expires_at supports bounded cleanup; (merchant_user_id, occurred_at DESC) supports tenant-safe investigation. Raw IP addresses, user agents, usernames attempted, passwords, hashes, and token values are prohibited.

analytics_access_audit_events

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
merchant_user_id	bigint required	restrictive FK to merchant_users
merchant_id	bigint required	part of composite ownership FK with merchant_user_id
event_type	text required	approved analytics/detail/export action code
result	text required	stable completed/rejected/aborted outcome code
period_id	text nullable	approved canonical period identifier
view_type	text nullable	approved detail/export view code
filter_fingerprint	bytea nullable	32-byte digest of canonical filters; never raw filter/cursor data
row_count	bigint nullable	non-negative completed export count only
byte_count	bigint nullable	non-negative completed export byte count only
request_id	text nullable	bounded safe correlation identifier
occurred_at	timestamptz required	action time
expires_at	timestamptz required	90-day retention deadline

The ownership FK guarantees that the user belongs to the recorded merchant. Checks enforce a 32-byte fingerprint when present and non-negative row/byte counts. Index expires_at supports automatic retention cleanup; (merchant_id, occurred_at DESC) and (merchant_user_id, occurred_at DESC) support operator-only investigation. Exported values, payer tokens, raw cursors/filters, credentials, user-controlled filenames, and peer data are prohibited. This table has no merchant-facing API or UI.

Payment facts

payment_sessions

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
source_session_key	text required	globally unique/non-empty opaque source key
merchant_id	bigint required	restrictive FK to merchants
terminal_id	bigint required	part of composite FK (terminal_id, merchant_id) to terminals
amount_rial	bigint required	positive session amount
fee_amount_rial	bigint required	non-negative real merchant fee, stored once
session_status	text required	non-empty source lifecycle code
created_at	timestamptz required	source Tehran-local value normalized to UTC
expires_at	timestamptz required	source expire_in normalized to UTC
source_origin	text required	bootstrap or zarinpal_live
import_run_id	bigint nullable	restrictive FK to import_runs

The origin check requires an import run for bootstrap and forbids one for zarinpal_live. Timestamp ordering is not enforced because source ordering anomalies are preserved as evidence rather than rewritten; expiry-before-creation, if found during exact validation, is reported according to the import specification.

Approved indexes:

• unique source_session_key
• (merchant_id, created_at DESC, id DESC) for period lists and keyset export
• (terminal_id, created_at DESC, id DESC) for terminal drill-down
• (merchant_id, session_status, created_at DESC, id DESC) for paid/unverified/failed detail views

payment_attempts

Column	Type/nullability	Constraint or meaning
id	bigint required	identity primary key
payment_session_id	bigint required	restrictive FK to payment_sessions
try_seq	smallint required	positive source sequence
try_status	text required	non-empty source lifecycle code; never NoAttempt
psp_id	bigint required	restrictive FK to psps
switch_response_code	text nullable	PSP-scoped raw code
issuer_bank_id	bigint nullable	restrictive FK to issuer_banks
payer_card_key	text nullable	unchanged restricted opaque token
init_time_ms	integer nullable	non-negative API duration
verify_time_ms	integer nullable	non-negative API duration
try_created_at	timestamptz required	source attempt creation normalized to UTC
verified_at	timestamptz nullable	preserved source instant
settled_at	timestamptz nullable	preserved source instant

Unique (payment_session_id, try_seq) is the attempt identity and lookup index. A check requires issuer_bank_id and payer_card_key to be either both null or both populated, matching the authoritative data guide and observed missingness. No timestamp-order check is used because documented anomalies are retained. No standalone payer-token, bank, PSP, or status index is added until an approved measured query justifies it.

Analytics invalidation

merchant_analytics_revisions

merchant_id bigint PK/FK to merchants; source_revision bigint required default 0 and non-negative; updated_at timestamptz required.

cohort_analytics_revisions

category_id bigint PK/FK to merchant_categories; source_revision bigint required default 0 and non-negative; updated_at timestamptz required.

Revision values are cache namespace markers, not transaction counts. They may advance more than once in one business operation without changing correctness.

Database-managed revision contract

Direct ZarinPal inserts may bypass Go, so invalidation cannot rely only on application callbacks. Versioned PostgreSQL statement-level AFTER INSERT triggers with transition tables update each affected merchant and category once per insert statement:

• session insert: use merchant/category relationships from the inserted sessions;
• attempt insert: join inserted attempts to their sessions, merchants, and categories;
• update/delete triggers do not exist in MVP because payment facts are append-only.

This is deliberate schema-owned logic, named in migrations and covered by integration tests. Bulk promotion remains efficient because a multi-row INSERT ... SELECT advances each affected revision once, not once per imported row. Trigger failure aborts the fact insert; analytics can never observe a new fact under an old revision.

Staging boundary

Importer staging tables are run-scoped, migration-owned technical structures rather than durable domain entities. They mirror raw columns plus import_run_id, source_row_number, and parse/validation metadata. They have no application Ent entity, no long-term API, and are truncated/dropped only by the explicit recovery lifecycle after the durable import report is complete. Promotion targets only the tables above.

Deletion, mutation, and retention

• Imported and live payment facts are append-only in MVP. Corrections require a new specification/ADR and matching invalidation behavior.
• Merchant/category/terminal/import provenance referenced by facts cannot be deleted. Synthetic display names and password hashes may be updated through approved identity workflows.
• Session rows are revoked/expired, not hard-deleted during their security lifetime. Authentication audit events expire after 30 days; analytics-access and CSV-export audit events expire after 90 days. Cleanup is allowed only after expires_at and must not remove payment/import facts.
• Redis analytics entries are disposable and expire after 30 days; no calculated analytics table exists in PostgreSQL.

Migration and verification requirements

• Atlas-generated versioned SQL is reviewed and committed; production startup never auto-migrates.
• Migration tests create the schema from zero and upgrade from every supported release state.
• Constraint tests cover terminal ownership, user/audit tenant ownership, no-attempt exclusion, attempt uniqueness, bootstrap/live provenance, paired card/bank nullability, session token digest length, audit fingerprint length, and revision-trigger atomicity.
• Query-plan verification uses production-shaped volumes and EXPLAIN (ANALYZE, BUFFERS) for approved analytics, detail pagination, CSV export, and session-authentication paths before adding further indexes.
• Initial fact tables are not partitioned. Partitioning requires measured operational need and a separate ADR.

Acceptance criteria

• The physical schema represents all 2,062,839 sessions and exactly 1,949,353 real attempts without a fake NoAttempt attempt.
• Every session's terminal is provably owned by the same merchant through a database foreign key.
• Every source session key and (session, try_seq) pair is unique after promotion.
• Session-level amount, fee, merchant, terminal, status, creation, expiry, origin, and provenance are stored once.
• verify_type is stored once per terminal and reconciles to every source record for that terminal.
• Payer tokens remain unchanged and restricted; they are absent from uniqueness constraints, logs, exports, peer results, and AI inputs.
• Direct session/attempt inserts atomically select new merchant/cohort revision namespaces.
• Approved period, authentication, drill-down, and export access paths have an explicit initial index or a documented measure-first rule.
• Schema creation, FK/check constraints, trigger behavior, 30/90-day retention cleanup, and query plans are integration-tested before implementation approval can be considered complete.

Boundaries

• Always: preserve source keys and nulls, enforce tenant ownership with FKs, store money at session grain, use UTC instants, version migrations, and test revision changes atomically.
• Ask first: add a table/column/index, denormalize tenant/category data, introduce partitioning, classify a new lifecycle status, or permit fact mutation/deletion.
• Never: create one wide CSV table, create a fake no-attempt row, duplicate session money on attempts, trust file order, store raw session tokens, or hide production migrations in startup.

Decision batch for approval

1. Store terminal-invariant verify_type on terminals, not every payment attempt.
2. Preserve future non-empty lifecycle status values as text and classify them through the metric catalogue instead of using rigid PostgreSQL enums.
3. Use explicit statement-level insert triggers for merchant/cohort analytics revisions because direct ZarinPal writes can bypass Go.
4. Persist minimal 30-day authentication audit events and 90-day analytics/export audit events because the approved Identity and API specs require queryable operator evidence without raw payloads, cursors, credentials, or sensitive data.
5. Approve the table, constraint, index, and restrictive-deletion contract above as the migration target.