zarinpal-analytics-backend/tasks/todo.md

ZarinPal Merchant Analytics Backend Task List

• Status: Approved — Phase 4 implementation authorized 2026-08-20
• Plan: tasks/plan.md
• Contract: docs/API-SPEC.md and docs/openapi.yaml version 1.0.0

Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. Foundation tasks use F identifiers; the previously approved API tasks retain their stable numeric identifiers.

Task F1: Establish the Go workspace and quality toolchain

Description: Create the thin command-root/modular-monolith skeleton, validated configuration boundary, structured logging baseline, and pinned repository quality commands without implementing product behavior.

Acceptance criteria:

• cmd/api, cmd/worker, cmd/importer, and migration/admin composition roots are thin and business packages live under internal/.
• Manual constructor wiring, context propagation, validated environment configuration, and safe structured logs follow approved ADRs/AGENTS.
• Pinned build/test/race/vet/lint/vulnerability/generation commands are documented and reproducible.

Verification: Run every scaffold quality command from a clean checkout; confirm no service starts implicitly and dependency/checksum diffs are approved.

Dependencies: Explicit implementation authorization.

Files likely touched: go.mod, go.sum, cmd/, internal/platform/, quality-tool configuration.

Estimated scope: Medium (split command roots if more than five files)

Task F2: Implement the normalized Ent model and migration baseline

Description: Translate the approved physical schema into Ent entities/relationships and reviewed Atlas SQL migrations, including identity, import, payment, calendar, and audit tables without cache-revision triggers.

Acceptance criteria:

• Tables, FKs, checks, unique/partial indexes, nullability, restrictive deletion, and 30/90-day audit retention match the physical schema.
• payment_sessions.import_run_id is required; NoAttempt never becomes an attempt; terminal verification method and session-grain money are stored once.
• Clean creation and supported upgrade migrations are versioned, reviewed, and never run from API startup.

Verification: Run zero-to-head/upgrade migration tests, schema introspection/ERD comparison, constraint negatives, and representative query-plan checks.

Dependencies: F1.

Files likely touched: internal/data/schema/, ent/schema/, migrations/, migration tests, Atlas configuration.

Estimated scope: Medium slices by schema group

Task F3: Implement pinned archive preflight and bounded staging

Description: Build the one-shot importer boundary that validates the fixed file/checksum/schema/resource caps and streams strict CSV/gzip records through pgx COPY into run-scoped staging.

Acceptance criteria:

• Fixed filename, SHA-256, 22-column schema, UTF-8/gzip integrity, byte/row/record caps, and regular read-only path checks fail closed.
• Parsing is bounded-memory, independent of file/session order, and records safe row numbers without logging source values.
• One PostgreSQL advisory lock prevents concurrent validate/import/recovery commands.

Verification: Unit/fuzz tests cover malformed CSV/UTF-8/gzip, overflow, traversal/symlink, expansion, cancellation, caps, and redaction; staging load stays within measured memory.

Dependencies: F1–F2.

Files likely touched: cmd/importer/, internal/importing/preflight/, internal/importing/staging/, focused tests.

Estimated scope: Medium, split preflight and staging commits

Task F4: Implement validation, atomic promotion, reports, and recovery

Description: Add set-based relationship validation, warning/fatal classification, exact reconciliation, transactional normalization, redacted reports, staging cleanup, no-op detection, and restart-from-source recovery.

Acceptance criteria:

• Any fatal issue promotes zero normalized facts; documented timestamp/sequence/null anomalies remain unchanged warnings.
• Pinned counts and per-merchant/status reconciliations pass before the sole import run becomes completed.
• Success/failure removes raw staging; interrupted recovery cleans only its run scope and restarts from the immutable archive.

Verification: Full-archive rehearsal plus relationship/rollback/fault-injection/concurrency/no-op/recovery/leak tests; JSON and Markdown reports reconcile.

Dependencies: F3.

Files likely touched: internal/importing/validation/, internal/importing/promotion/, internal/importing/report/, recovery command/tests.

Estimated scope: Medium slices; never one XL commit

Task F5: Implement synthetic merchant profiles and demo-account seeding

Description: Generate deterministic non-sensitive merchant display profiles and exactly one initial demo user per merchant through a separate secure administrative flow.

Acceptance criteria:

• Display names are deterministic, clearly synthetic in storage, and stable across identical imports.
• Usernames are unique and credentials use approved Argon2id hashing; plaintext is emitted once only to an explicit protected output.
• Re-running the seed is idempotent and never derives passwords from merchant/source keys.

Verification: Determinism/idempotency/hash/permission/redaction tests and two-merchant credential smoke fixture.

Dependencies: F2 and completed F4 provider behavior.

Files likely touched: cmd/admin/, internal/identity/seeding/, profile generator/tests, operator output adapter.

Estimated scope: Medium (3–5 files)

Task F6: Implement password authentication and server-side sessions

Description: Implement demo credential verification, generic throttled outcomes, two-active-session enforcement, opaque token digests, CSRF binding, expiry/activity coalescing, logout/revocation, and authentication audit retention.

Acceptance criteria:

• Authentication never reveals account existence and returns tenant context only from a valid server-side session.
• Cookie/token/CSRF, idle/absolute expiry, revocation, disabled account, and two-session behavior match Identity spec.
• Raw credentials/tokens/IP/user-agent values are absent from storage, logs, and audit evidence.

Verification: Table/race/integration tests cover enumeration, throttles, fixation, rotation, expiry, revocation, concurrency, and 30-day cleanup.

Dependencies: F2 and F5.

Files likely touched: internal/identity/, session repository, throttling/audit adapters, focused tests.

Estimated scope: Medium slices

Task F7: Implement reporting periods and deterministic metric queries

Description: Build calendar-period discovery and SQL/Ent query services for catalogue 0.4.0, preserving session grain, missingness, Saturday–Friday/Jalali/season boundaries, and peer suppression/equal weighting.

Acceptance criteria:

• Every catalogue metric reconciles for day/week/month/season and emits unit, denominator/sample/coverage, dataset snapshot, metric version, and calculation time.
• Paid plus Verified, verification completion, retry recovery, timing percentiles, fees, and NoAttempt semantics match approved formulas.
• Peer outputs exclude the active merchant, weight merchants equally, invert lower-is-better percentiles, and suppress below both thresholds.

Verification: Independent SQL oracle tests, boundary fixtures, large/skewed merchant fixtures, null/zero/unknown-status tests, and measured query plans.

Dependencies: F2 and trusted F4 dataset fixtures.

Files likely touched: internal/analytics/periods/, internal/analytics/metrics/, peer queries, fixtures/tests.

Estimated scope: Medium slices by section

Task F8: Implement Redis/Asynq analytics run orchestration

Description: Implement idempotent four-section runs, dataset-versioned 30-day cache keys, Asynq v0.26.0 jobs, bounded concurrency, retries, rate limits, event history, and sibling-failure isolation.

Acceptance criteria:

• Equivalent starts single-flight; all sections begin in bounded parallel and publish ready results atomically/independently.
• Attempts time out at 60 seconds and only transient failures receive two exponential-backoff/jitter retries.
• Redis loss yields safe cold recomputation; one-hour events/15-second heartbeat and independent user/merchant limits match specs.

Verification: Queue/cache integration, race, failure, retry timing, Redis-loss, deduplication, replay, rate-limit, and sensitive-payload tests.

Dependencies: F1, F6, and F7.

Files likely touched: internal/analytics/runs/, internal/analytics/jobs/, Redis adapters, worker root/tests.

Estimated scope: Medium slices

Task F9: Implement tenant-scoped detail and export providers

Description: Build the shared authorized query/projection for unverified-paid, failed, and retried details, deterministic keyset pagination, export preflight/count/stream inputs, and operator audit metadata.

Acceptance criteria:

• Every query requires session-derived merchant scope and approved period/filter allowlists.
• Detail/export projections reconcile to metrics and exclude payer tokens, peer rows, credentials, and internal keys.
• Stable ordering, cursor inputs, 100,000-row/50-MiB/60-second export caps, and audit metadata are available to the API adapter.

Verification: Tenant/tampering/reconciliation/pagination/snapshot/cap/query-plan/leak tests against PostgreSQL.

Dependencies: F2, F6, and F7.

Files likely touched: internal/analytics/details/, cursor/projection package, export preflight, audit/tests.

Estimated scope: Medium slices

Task F10: Stabilize provider interfaces and backend integration

Description: Wire Identity, period, analytics-run/event, detail/export, audit, PostgreSQL, and Redis providers behind narrow interfaces consumed by the Merchant API.

Acceptance criteria:

• Composition roots wire explicit constructors with bounded pools/timeouts and graceful lifecycle.
• Provider contracts match OpenAPI semantics without importing HTTP types into domain packages.
• A non-HTTP integration harness completes seed/login/period/run/partial-result/detail/export-provider flows for two isolated merchants.

Verification: Full provider integration, race, shutdown, dependency-loss, tenant-boundary, and safe-log tests.

Dependencies: F4–F9.

Files likely touched: provider interfaces/adapters, cmd/api/, cmd/worker/, integration tests.

Estimated scope: Medium (split wiring and integration)

Checkpoint F: Backend provider readiness

• Tasks F1–F10 and every foundation acceptance criterion pass.
• Full pinned import, two-merchant identity, metric reconciliation, Redis-loss, and tenant isolation evidence is reviewed.
• Human approves provider readiness before completing HTTP feature slices.

Task 1: Establish the OpenAPI toolchain and conformance harness

Description: Select, approve, and pin the Go OpenAPI generation/validation tooling; make contract validation and generated-artifact drift checks reproducible from a clean checkout.

Acceptance criteria:

• One approved pinned toolchain validates OpenAPI 3.1 and produces only the intended server/types artifacts.
• Generated artifacts contain no hand-edited business logic and regenerate with a clean Git diff.
• Stable operationId values, schemas, media types, and security requirements are covered by a contract test harness.

Verification:

• Run the documented OpenAPI validate/generate/check commands from a clean checkout.
• Run focused generator/conformance tests and confirm a deliberate contract mismatch fails.
• Review generated and lockfile/checksum diffs; approve any new dependency before installation.

Dependencies: Explicit implementation authorization and F1.

Files likely touched:

• go.mod
• go.sum
• internal/api/contract/
• internal/api/contract_test.go
• approved command/tool configuration

Estimated scope: Medium (3–5 files/artifact groups)

Task 2: Build the shared HTTP boundary and protected-request middleware

Description: Create the API server boundary for /api/v1, central Problem Details mapping, request ids, 16 KiB body/8 KiB target limits, no-store/security headers, session resolution, and CSRF/origin enforcement.

Acceptance criteria:

• All protected handlers receive merchant/user context only from a verified server-side session.
• Boundary failures use the approved RFC 9457 schema/code/status mapping without internal details.
• Request limits, unknown fields/parameters, CSRF/origin checks, and no-store/security headers behave as specified.

Verification:

• Focused middleware tests cover missing/expired/revoked sessions, malformed input, size limits, request ids, CSRF, and origin/fetch metadata.
• Cross-tenant client inputs cannot create or replace merchant context.
• go test ./... and go vet ./... pass.

Dependencies: Task 1 and F6 provider interface.

Files likely touched:

• internal/api/http/server.go
• internal/api/http/problem.go
• internal/api/http/middleware.go
• internal/api/http/middleware_test.go

Estimated scope: Medium (4 files)

Checkpoint 1: Boundary readiness

• Tasks 1–2 acceptance criteria pass.
• OpenAPI generation is reproducible with a clean diff.
• Human review confirms the tenant and error boundaries before feature handlers begin.

Task 3: Implement authentication session endpoints

Description: Implement login, current-session, and logout as one vertical slice through the approved Identity provider, including secure cookie and generic-failure behavior.

Acceptance criteria:

• Login creates a revocable session and sets only the approved __Host-zpa_session cookie attributes.
• Current-session JSON matches OpenAPI and exposes a session-bound CSRF token but no authentication token.
• Logout revokes server state before clearing the cookie; expected credential failures remain indistinguishable and throttled.

Verification:

• Integration tests cover success, enumeration resistance, cookie flags, rotation, expiry, revocation, throttling, CSRF, and dependency failure.
• OpenAPI conformance tests cover all auth statuses and headers.
• go test -race ./... passes for affected packages.

Dependencies: Task 2 and F6.

Files likely touched:

• internal/api/http/auth_handler.go
• internal/api/http/auth_handler_test.go
• internal/api/identity.go
• cmd/api/main.go

Estimated scope: Medium (4 files)

Task 4: Implement approved reporting-period discovery

Description: Return only allowlisted day/week/Jalali-month/Persian-season periods with exact UTC/Tehran boundaries, ready Persian labels, and the approved default.

Acceptance criteria:

• Period responses match OpenAPI and use backend-calculated half-open boundaries.
• Iranian weeks start Saturday; default selection is the latest complete Jalali month containing source data.
• Unknown parameters/types are rejected and arbitrary date ranges are impossible.

Verification:

• Table tests cover all period types, month/year boundaries, Saturday–Friday weeks, empty source ranges, and timezone independence.
• Integration tests reconcile returned ids/boundaries with calendar_dates.
• OpenAPI examples and go test ./... pass.

Dependencies: Task 2 and F7.

Files likely touched:

• internal/api/http/period_handler.go
• internal/api/http/period_handler_test.go
• internal/api/periods.go

Estimated scope: Medium (3 files)

Task 5: Implement analytics-run start and snapshot retrieval

Description: Implement idempotent run creation/current-run resolution and tenant-scoped snapshot retrieval for all four sections.

Acceptance criteria:

• Same intent/key replays safely, key/body conflicts fail, and equivalent concurrent starts resolve one current run.
• Run snapshots serialize every section state/result exactly as OpenAPI defines.
• Missing and cross-tenant run ids return the same 404 RESOURCE_NOT_FOUND response.

Verification:

• Concurrency tests cover atomic idempotency claim, in-flight duplicates, the immutable dataset snapshot, and no duplicate jobs.
• Contract tests cover 200, 202, 401, 403, 404, 422, 429, and dependency failures.
• Integration tests assert that rejected/limited requests enqueue no work.

Dependencies: Tasks 1–2 and F8.

Files likely touched:

• internal/api/http/run_handler.go
• internal/api/http/run_handler_test.go
• internal/api/analytics.go
• internal/api/idempotency.go

Estimated scope: Medium (4 files)

Checkpoint 2: Core API flow

• Tasks 3–5 acceptance criteria pass.
• Authenticated merchant flow works through the actual HTTP server and test dependencies.
• Cross-tenant and idempotency adversarial tests pass.
• Human review approves the core flow before streaming/detail work.

Task 6: Implement authenticated SSE delivery and recovery

Description: Stream typed run/section events with 15-second heartbeats, one-hour authorized replay, snapshot reset, and session-aware disconnect/reconnect behavior.

Acceptance criteria:

• Event framing, ids, names, and JSON data match the approved SSE schemas and UTF-8 media type.
• Retained ids replay only later authorized events; unavailable ids reset with run.snapshot and never disclose another tenant.
• Session expiry/revocation, connection limits, and completed-run closure behave as specified.

Verification:

• Integration tests cover ordered partial readiness, failure isolation, suppression, heartbeat timing, replay, reset, disconnect, revocation, and user/merchant connection limits.
• Proxy-style tests assert no buffering/compression and required stream headers.
• Race tests cover concurrent publish/replay/disconnect paths.

Dependencies: Task 5 and F8.

Files likely touched:

• internal/api/http/sse_handler.go
• internal/api/http/sse_handler_test.go
• internal/api/events.go
• internal/api/events_test.go

Estimated scope: Medium (4 files)

Task 7: Implement exhausted-section manual retry

Description: Add the idempotent retry-attempt subresource for one exhausted failed section without disturbing ready siblings.

Acceptance criteria:

• Only an exhausted, retryable failed section for the current run/dataset snapshot is accepted.
• Retry idempotency and independent user/merchant throttles prevent duplicate work.
• Ready siblings remain unchanged and stale/cross-tenant requests fail safely.

Verification:

• Tests cover retryable/non-retryable states, duplicate keys, stale dataset snapshots, throttling, cross-tenant ids, and no sibling restart.
• Rejected retries enqueue no work.
• OpenAPI conformance and go test -race ./... pass.

Dependencies: Tasks 5–6.

Files likely touched:

• internal/api/http/retry_handler.go
• internal/api/http/retry_handler_test.go
• internal/api/retry.go

Estimated scope: Medium (3 files)

Checkpoint 3: Asynchronous behavior

• Tasks 6–7 acceptance criteria pass.
• SSE and retry runtime checks pass through the actual queue/cache test boundary.
• Human review confirms recovery and rate-limit behavior.

Task 8: Implement cursor-paginated session detail views

Description: Implement the three approved detail views with deterministic ordering, opaque bound cursors, default 25/maximum 100 rows, and a strict field allowlist.

Acceptance criteria:

• UNVERIFIED_PAID, FAILED, and RETRIED filters match catalogue definitions and reconcile to aggregates.
• Cursors bind tenant, canonical filters, ordering, expiry, and last row; invalid reuse returns generic INVALID_CURSOR.
• Detail rows expose only approved fields and the tenant-scoped opaque terminal reference.

Verification:

• Tests cover page boundaries, duplicate timestamps, cursor tampering/expiry/filter change/cross-tenant reuse, limits, and empty results.
• Integration queries reconcile representative merchants and approved periods.
• Sensitive-field absence and measured query-plan checks pass.

Dependencies: Tasks 2 and 4 plus F9.

Files likely touched:

• internal/api/http/detail_handler.go
• internal/api/http/detail_handler_test.go
• internal/api/details.go
• internal/api/cursor.go
• internal/api/cursor_test.go

Estimated scope: Medium (5 files)

Task 9: Implement safeguarded Persian CSV export

Description: Stream the same authorized detail predicate/projection as Persian CSV with BOM, Persian spaced headers/statuses, machine-safe numeric/ISO values, formula neutralization, and approved caps.

Acceptance criteria:

• CSV and JSON share one canonical tenant/filter/projection policy; CSV adds no unavailable field.
• Output matches the approved encoding, headers, labels, quoting, disposition, no-store, and nosniff contract.
• Preflight rejects over 100,000 rows/50 MiB, generation ends within 60 seconds, throttling applies, and interrupted streams are audited as incomplete.

Verification:

• Golden tests cover Persian output, BOM, CRLF, quotes/newlines, all status mappings, and stable headers with spaces.
• Security tests cover =, +, -, @, tab, CR, LF formula prefixes and forbidden-field absence.
• Integration tests cover filter parity, snapshot consistency, caps, deadline, disconnect, throttle, and content headers.

Dependencies: Task 8.

Files likely touched:

• internal/api/http/export_handler.go
• internal/api/http/export_handler_test.go
• internal/api/export.go
• internal/api/export_test.go

Estimated scope: Medium (4 files)

Checkpoint 4: Reconciliation and data boundary

• Tasks 8–9 acceptance criteria pass.
• Detail/CSV reconciliation and sensitive-data negative tests pass.
• Export resource limits hold under representative merchant volume.
• Human review approves the merchant evidence boundary.

Task 10: Publish the stable API problem-type catalogue

Description: Document every stable application error code and make each approved https://zarinpal.amiirkhl.ir/docs/errors/{error-slug} URI resolve to safe human-readable guidance.

Acceptance criteria:

• Every OpenAPI problem code maps to one stable slug, title, status, meaning, safe resolution guidance, and retry rule.
• Published pages expose no internal topology, SQL, stack, secret, tenant existence, or security-control detail useful for abuse.
• Contract tests fail when a returned problem type/code lacks catalogue documentation.

Verification:

• Link checks confirm every production type URI resolves successfully.
• OpenAPI and catalogue mapping tests pass with no duplicate slug/code.
• Security review approves public wording.

Dependencies: Tasks 2–9 for the final error-code set; hosting mechanism approval.

Files likely touched:

• docs/ERROR-CATALOGUE.md
• error-page source/template location approved by the hosting plan
• internal/api/http/problem_catalogue_test.go

Estimated scope: Medium (2–3 files/artifact groups)

Task 11: Add API audit and observability instrumentation

Description: Instrument the approved audit events, metrics, logs, and traces with bounded-cardinality safe metadata and 90-day analytics/export-audit retention.

Acceptance criteria:

• Required auth/run/SSE/detail/export audit outcomes and operational metrics are emitted with correlation ids.
• Only authorized backend operators can access analytics/export audit records; automatic deletion occurs after 90 days.
• Telemetry excludes credentials, tokens, raw cursors/filters, payment rows, payer tokens, peer data, and high-cardinality sensitive labels.

Verification:

• Tests assert required event/metric emission for success, rejection, failure, timeout, and interrupted export.
• Negative leak tests inspect structured logs, traces, and audit records.
• Retention/access-control tests pass using an accelerated test clock.

Dependencies: Tasks 3–10 and the approved audit/observability adapters.

Files likely touched:

• internal/api/observability.go
• internal/api/audit.go
• internal/api/observability_test.go
• internal/api/audit_test.go

Estimated scope: Medium (4 files)

Task 12: Complete contract, security, race, and performance verification

Description: Run and close the full implementation Definition of Done across all API operations and critical failure/abuse paths.

Acceptance criteria:

• Every OpenAPI operation/status/media type/header/security requirement has automated contract coverage.
• Tenant isolation, session/CSRF, idempotency, SSE, cursor, export, audit, and sensitive-field abuse suites pass.
• Representative detail/export query plans and resource bounds meet approved behavior with no unreviewed exception.

Verification:

• Run go test ./..., go test -race ./..., go vet ./..., golangci-lint run, and govulncheck ./....
• Run reproducible OpenAPI validation/generation and confirm a clean Git diff.
• Exercise critical merchant flows end to end through the deployed-style proxy/test environment.

Dependencies: Tasks 1–11.

Files likely touched:

• API contract/integration test packages
• approved test fixtures containing no sensitive source data
• API runbook/documentation corrections if verification finds drift

Estimated scope: Medium (3–5 files/artifact groups)

Checkpoint 5: Complete

• Tasks 1–12 and every checkpoint pass.
• Full project Definition of Done is satisfied.
• No unrelated frontend, deployment, physical-schema, ERD, or raw-data changes are included.
• Human review approves completion before merge or deployment.

Approved redesign API increment — 2026-08-20

• Task 13: Return every complete dataset-backed DAY, WEEK, JALALI_MONTH, and PERSIAN_SEASON period with the latest complete Jalali month as default.
  • Acceptance: pure boundary tests cover Nowruz, month rollover, leap Esfand, Iranian Saturday–Friday weeks, and dataset boundaries.
  • Verify: focused analytics tests and go test ./....
• Task 14: Add authenticated merchant terminal discovery and terminal-scoped analytics runs.
  • Acceptance: unknown or other-merchant terminals are rejected without changing merchant scope; an empty terminal means all merchant terminals.
  • Verify: service/handler tests plus local authenticated smoke.
• Task 15: Add bounded daily payment-history points to overview data.
  • Acceptance: each point contains date, bank-paid amount, bank-paid sessions, and total sessions for the selected period/scope.
  • Verify: query reconciliation against aggregate metrics.
• Task 16: Add the ALL transaction view, status and amount filters, and a merchant-owned transaction detail resource.
  • Acceptance: filters are allowlisted/parameterized, pagination happens after filters, and attempts are ordered without payer tokens.
  • Verify: focused tests, local drill-down smoke, and safe 404 check.
• Task 17: Update OpenAPI and run the complete backend gate.
  • Acceptance: generated frontend types compile and the feature API runs beside the current API on port 8081.
  • Verify: go test ./..., go vet ./..., build, and authenticated HTTP smoke.