zarinpal-analytics-backend/docs/ANALYTICS-SPEC.md

Analytics Engine Capability Spec

• Status: Approved specification — implementation not yet authorized
• Capability: Analytics engine
• Depends on: Approved Data Foundation specification
• Last updated: 2026-08-20

Objective

Turn normalized ZarinPal payment facts into trustworthy, merchant-scoped analytics that explain payment performance, verification gaps, retry outcomes, processing time, fees, trends, and same-category position. Deterministic calculations are the product source of truth; later AI assistance may explain or recommend actions from sanitized aggregates but cannot define or alter metrics.

Actors and consumers

• Authenticated merchant: sees detailed metrics and drill-downs only for its merchant boundary.
• Merchant API: requests versioned analytics sections and exposes typed results/events.
• Dashboard: renders sections independently as they become ready.
• Analytics worker: calculates idempotent section results from PostgreSQL and caches approved aggregates in Redis.
• Future AI adapter: post-MVP consumer of minimized, approved aggregate facts only.

In scope

• Daily, Saturday–Friday weekly, Jalali-month, and Persian-season analytics in Asia/Tehran
• Overview, payment performance, fees, and peer-comparison sections
• Payment success and volume with correct Paid/Verified semantics
• Verification completion and unverified-paid gaps
• Retry recovery and attempt/session diagnostics
• Processing-time p50/p95 with sample and coverage
• Real merchant fee amounts and paid fee rate at session grain
• Current-period trends against an approved equivalent baseline
• Same-category merchant-weighted peer median and performance percentile
• Lazy, bounded parallel calculation with partial SSE delivery
• Versioned 30-day Redis cache and PostgreSQL revision invalidation
• Traceability from aggregate metrics to permitted merchant-owned evidence

Out of scope for MVP

• Holiday or named-event reporting
• AI-generated recommendations or free-form LLM analysis
• Forecasting, causal claims, or automated campaign execution
• Arbitrary merchant-created formulas or cross-merchant filters
• Real-time streaming payment ingestion owned by this service
• Persisting calculated analytics as PostgreSQL business facts
• Public or unauthenticated analytics

Canonical sections

Section id	Responsibility	Minimum result
overview	Merchant-level outcome and trend summary	bank-paid sessions/amount, success rate, verification completion, selected period, prior-period comparison
payment_performance	Status, retry, and timing analysis	paid/verified/unverified/failed/reversed counts, retry recovery, init/verify p50/p95 with coverage
fees	Merchant fee analysis	paid fee amount, paid amount denominator, paid fee rate, period trend
peer_comparison	Privacy-bounded same-category benchmark	eligible peer count/session count, cohort median, direction-aware merchant percentile or typed suppression

Every section uses a versioned typed contract and has independent queued, calculating, ready, failed, and where applicable suppressed states.

Metric and period contract

• METRIC-CATALOGUE.md version 0.4.0 is canonical for formulas, units, grains, missing-data behavior, and peer direction.
• Paid and Verified both mean bank payment succeeded; only Verified is finalized. Never label Paid as failure.
• Amount and fee calculations count each eligible payment session once. Attempt rows cannot multiply session values.
• Rates are calculated from summed numerator and denominator components for the selected period, never averaged from daily percentages.
• Timing percentiles exclude null values and expose sample size and non-null coverage; missing time never becomes zero.
• NoAttempt markers create zero-attempt sessions and never count as attempts or retries. InBank is an observed attempt stage, not proof of rejection.
• Every result states the exact Tehran-local half-open period, Jalali label, unit, sample/denominator, source revision, metric-definition version, and calculation timestamp.
• Approved period types are one Tehran-local day, one Saturday–Friday week, one Jalali month, or one Persian season. Holiday/event dimensions do not exist in the MVP contract.

Peer comparison contract

• Eligible peers are other merchants in the active merchant's category for the same selected period.
• Suppress the peer result unless at least 10 other merchants and 1,000 combined peer sessions contribute.
• Calculate each peer merchant's metric independently and take the median of those merchant values; do not transaction-weight the cohort.
• Return the cohort median and active merchant's performance percentile. Higher displayed percentile always means better, so processing-time and fee-rate ranking is inverted.
• Benchmark payment success, verification completion, retry recovery, init/verify p50/p95, and paid fee rate.
• Never return peer identities, individual values/ranks, rows, or arbitrary filters that enable differencing attacks.

Calculation and cache lifecycle

1. An idempotent authenticated start request resolves merchant, selected period, merchant revision, cohort revision, and metric-definition version.
2. Equivalent requests map to one current analytics run and do not enqueue duplicate work.
3. Missing section results are enqueued together and calculated independently by the Go worker through Asynq v0.26.0, explicitly a pre-1.0 dependency pinned for compatibility.
4. Worker limits default to 16 jobs globally and four per merchant. All four core sections begin in bounded parallel and become visible independently.
5. Each attempt times out after 60 seconds. Transient failures receive at most two retries using exponential backoff of approximately one then two seconds plus jitter; deterministic failures are not retried.
6. Successful section results are published atomically to Redis under revisioned/versioned keys with a 30-day TTL. PostgreSQL facts and revision rows remain authoritative.
7. Relevant fact inserts advance the affected merchant and category/cohort revisions in PostgreSQL. New requests use new keys; old keys expire without scans.
8. Redis loss causes safe lazy recomputation. It never causes loss of payment facts or metric definitions.

Partial delivery contract

• A separate authenticated SSE connection streams run/section state and each completed section immediately.
• SSE sends a heartbeat every 15 seconds and retains authorized replay events for one hour.
• Reconnect uses the last event id where retained; otherwise the client fetches the current run snapshot and reopens the stream.
• One failed section never hides ready siblings. Manual retry targets only an exhausted failed section and cannot clear successful results.
• Start limits are 10 requests/minute with burst 3; manual retry limits are 3 requests/10 minutes with burst 1; SSE limits are two concurrent connections/user and four/merchant. Independent user and merchant buckets return 429 with Retry-After before work is queued.

Traceability and privacy

• Merchant-owned aggregate cards link to definitions and, where approved, tenant-scoped session/attempt evidence that reconciles to the aggregate.
• Payer-card tokens remain internal, opaque, merchant-scoped inputs and never appear in APIs, logs, exports, caches, peer results, or AI payloads.
• Redis values contain approved aggregates and trace metadata only, not raw payment rows or credentials.
• Cross-merchant calculations execute server-side and return only approved aggregate/suppressed contracts.
• Future AvalAI prompts may receive minimized aggregates only after a separate AI capability approval.

Failure and edge behavior

• Zero, undefined denominator, unavailable source, peer suppression, calculation failure, and not-yet-ready are distinct typed states.
• Unknown future payment statuses are excluded from success numerator/denominator and reported through coverage evidence until classified.
• Reversed remains outside success and is displayed separately when non-zero.
• Empty approved periods return a valid empty result, not fabricated zero rates or a system failure.
• Section publication is atomic; partial/failed calculations never overwrite a valid ready cache entry.
• Database, Redis, queue, timeout, and authorization errors are distinguishable internally while merchant-facing errors remain safe and actionable.

Actionable insight threshold

• A deterministic actionable insight is eligible only when its comparison contains at least 100 relevant payment sessions.
• Rate insights require an absolute change of at least 3 percentage points. Amount and count insights require an absolute relative change of at least 20 percent.
• The threshold is a display eligibility rule, not a change to the underlying metric. Exact metrics and comparisons remain available even when no insight card is emitted.
• A zero, missing, or undefined baseline cannot produce a relative-change insight. Return the metric and an explicit unavailable comparison instead of dividing by zero or inventing a trend.
• Meeting a threshold does not prove causation. Insight language describes the observed change and a possible merchant action without claiming why it happened.

Observability

• Measure section queue delay, calculation duration, query duration, cache hit/miss, retry count, failure class, SSE delivery/reconnect, and suppression rate.
• Correlate start request, analytics run, section job, cache key version, and SSE event without logging sensitive merchant rows or payer tokens.
• Record PostgreSQL query plans during performance verification and add indexes only from measured access paths.
• Define service-level objectives and alert thresholds in the Platform Operations spec, not implicitly in worker code.

Acceptance criteria

• Every metric reconciles to an independent PostgreSQL query over normalized facts for representative merchants and all approved period types.
• Session-grain amount/fee totals remain unchanged when a session has multiple attempts.
• Paid plus Verified success, verification completion, retry recovery, and timing coverage match catalogue definitions.
• Saturday–Friday, Jalali-month, and Persian-season boundaries match calendar_dates and API half-open ranges.
• First access starts all missing sections once; ready sections stream independently and equivalent concurrent starts deduplicate.
• Cache hits avoid repeating source aggregation; merchant/cohort revision changes select fresh keys and recompute only missing sections.
• Redis flush causes correct cold-cache rebuilding without source-data loss.
• Timeouts, transient retry limits/backoff, failed-section isolation, manual retry, rate limits, heartbeat, replay, and snapshot fallback behave as specified.
• Peer results exclude the active merchant, use equal merchant weighting, obey both suppression thresholds, and never expose individual peers.
• Tenant-bound drill-downs reconcile to aggregates and cannot expose another merchant through parameter manipulation.
• Read-only drill-downs cover unverified-paid, failed, and retried sessions. Failed-session filters may distinguish zero-attempt and reached-bank stages without exposing payer tokens or claiming undocumented causes.
• No raw payer token, credential, cross-tenant row, or AvalAI secret enters results, logs, cache payloads, or browser data.

Boundaries

• Always: use catalogue definitions, preserve grain, enforce tenant scope, expose period/unit/coverage/version, calculate idempotently, and keep results reproducible.
• Ask first: add/change a metric, period, peer cohort, threshold, cache key/version, drill-down, AI input, or cross-service interface.
• Never: average rates, multiply session money by attempts, classify Paid as failure, expose peer rows, trust client tenant ids, or let AI become the metric source of truth.

Resolved decisions

• Normalized PostgreSQL facts are authoritative; Redis is a 30-day disposable analytics cache.
• Four sections calculate lazily and in bounded parallel, with partial authenticated SSE delivery.
• Reporting covers day, Iranian week, Jalali month, and Persian season only.
• Peer comparison uses same-category merchant-weighted medians and direction-aware percentiles with the approved suppression threshold.
• Payment lifecycle, retry, timing, and real-fee definitions follow METRIC-CATALOGUE.md and ADR 0013.
• Asynq v0.26.0 is pinned and acknowledged as pre-1.0; jobs have bounded concurrency, timeouts, rate limits, and exponential-backoff retries.
• Default first view is the latest complete Jalali month containing source data, avoiding an empty current month in the static demo.
• Compare with the immediately previous equivalent complete period. If it is empty or unavailable, return an explicit unavailable comparison rather than silently substituting another baseline.
• Overview priority is bank-paid amount, bank-paid session count, payment success rate, then verification completion. Retry, timing, and fee analysis follow below.
• MVP read-only drill-downs cover unverified-paid, failed, and retried sessions; payer tokens and peer-level rows are excluded.
• Actionable insight cards require at least 100 relevant sessions and either a 3-percentage-point rate change or a 20-percent amount/count change. Below-threshold comparisons remain visible as metrics but do not generate an action claim.

Open product decisions before implementation planning

• None. Any new or changed metric, period, cohort, threshold, or cross-service contract requires a specification amendment before implementation.