zarinpal-analytics-backend/tasks/todo.md

# ZarinPal Merchant Analytics Backend Task List

- Status: Approved — Phase 4 implementation authorized 2026-08-20
- Plan: `tasks/plan.md`
- Contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0`

Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. Foundation tasks use `F` identifiers; the previously approved API tasks retain their stable numeric identifiers.

## Task F1: Establish the Go workspace and quality toolchain

**Description:** Create the thin command-root/modular-monolith skeleton, validated configuration boundary, structured logging baseline, and pinned repository quality commands without implementing product behavior.

**Acceptance criteria:**

- [ ] `cmd/api`, `cmd/worker`, `cmd/importer`, and migration/admin composition roots are thin and business packages live under `internal/`.
- [ ] Manual constructor wiring, context propagation, validated environment configuration, and safe structured logs follow approved ADRs/AGENTS.
- [ ] Pinned build/test/race/vet/lint/vulnerability/generation commands are documented and reproducible.

**Verification:** Run every scaffold quality command from a clean checkout; confirm no service starts implicitly and dependency/checksum diffs are approved.

**Dependencies:** Explicit implementation authorization.

**Files likely touched:** `go.mod`, `go.sum`, `cmd/`, `internal/platform/`, quality-tool configuration.

**Estimated scope:** Medium (split command roots if more than five files)

## Task F2: Implement the normalized Ent model and migration baseline

**Description:** Translate the approved physical schema into Ent entities/relationships and reviewed Atlas SQL migrations, including identity, import, payment, calendar, and audit tables without cache-revision triggers.

**Acceptance criteria:**

- [ ] Tables, FKs, checks, unique/partial indexes, nullability, restrictive deletion, and 30/90-day audit retention match the physical schema.
- [ ] `payment_sessions.import_run_id` is required; `NoAttempt` never becomes an attempt; terminal verification method and session-grain money are stored once.
- [ ] Clean creation and supported upgrade migrations are versioned, reviewed, and never run from API startup.

**Verification:** Run zero-to-head/upgrade migration tests, schema introspection/ERD comparison, constraint negatives, and representative query-plan checks.

**Dependencies:** F1.

**Files likely touched:** `internal/data/schema/`, `ent/schema/`, `migrations/`, migration tests, Atlas configuration.

**Estimated scope:** Medium slices by schema group

## Task F3: Implement pinned archive preflight and bounded staging

**Description:** Build the one-shot importer boundary that validates the fixed file/checksum/schema/resource caps and streams strict CSV/gzip records through pgx `COPY` into run-scoped staging.

**Acceptance criteria:**

- [ ] Fixed filename, SHA-256, 22-column schema, UTF-8/gzip integrity, byte/row/record caps, and regular read-only path checks fail closed.
- [ ] Parsing is bounded-memory, independent of file/session order, and records safe row numbers without logging source values.
- [ ] One PostgreSQL advisory lock prevents concurrent validate/import/recovery commands.

**Verification:** Unit/fuzz tests cover malformed CSV/UTF-8/gzip, overflow, traversal/symlink, expansion, cancellation, caps, and redaction; staging load stays within measured memory.

**Dependencies:** F1–F2.

**Files likely touched:** `cmd/importer/`, `internal/importing/preflight/`, `internal/importing/staging/`, focused tests.

**Estimated scope:** Medium, split preflight and staging commits

## Task F4: Implement validation, atomic promotion, reports, and recovery

**Description:** Add set-based relationship validation, warning/fatal classification, exact reconciliation, transactional normalization, redacted reports, staging cleanup, no-op detection, and restart-from-source recovery.

**Acceptance criteria:**

- [ ] Any fatal issue promotes zero normalized facts; documented timestamp/sequence/null anomalies remain unchanged warnings.
- [ ] Pinned counts and per-merchant/status reconciliations pass before the sole import run becomes completed.
- [ ] Success/failure removes raw staging; interrupted recovery cleans only its run scope and restarts from the immutable archive.

**Verification:** Full-archive rehearsal plus relationship/rollback/fault-injection/concurrency/no-op/recovery/leak tests; JSON and Markdown reports reconcile.

**Dependencies:** F3.

**Files likely touched:** `internal/importing/validation/`, `internal/importing/promotion/`, `internal/importing/report/`, recovery command/tests.

**Estimated scope:** Medium slices; never one XL commit

## Task F5: Implement synthetic merchant profiles and demo-account seeding

**Description:** Generate deterministic non-sensitive merchant display profiles and exactly one initial demo user per merchant through a separate secure administrative flow.

**Acceptance criteria:**

- [ ] Display names are deterministic, clearly synthetic in storage, and stable across identical imports.
- [ ] Usernames are unique and credentials use approved Argon2id hashing; plaintext is emitted once only to an explicit protected output.
- [ ] Re-running the seed is idempotent and never derives passwords from merchant/source keys.

**Verification:** Determinism/idempotency/hash/permission/redaction tests and two-merchant credential smoke fixture.

**Dependencies:** F2 and completed F4 provider behavior.

**Files likely touched:** `cmd/admin/`, `internal/identity/seeding/`, profile generator/tests, operator output adapter.

**Estimated scope:** Medium (3–5 files)

## Task F6: Implement password authentication and server-side sessions

**Description:** Implement demo credential verification, generic throttled outcomes, two-active-session enforcement, opaque token digests, CSRF binding, expiry/activity coalescing, logout/revocation, and authentication audit retention.

**Acceptance criteria:**

- [ ] Authentication never reveals account existence and returns tenant context only from a valid server-side session.
- [ ] Cookie/token/CSRF, idle/absolute expiry, revocation, disabled account, and two-session behavior match Identity spec.
- [ ] Raw credentials/tokens/IP/user-agent values are absent from storage, logs, and audit evidence.

**Verification:** Table/race/integration tests cover enumeration, throttles, fixation, rotation, expiry, revocation, concurrency, and 30-day cleanup.

**Dependencies:** F2 and F5.

**Files likely touched:** `internal/identity/`, session repository, throttling/audit adapters, focused tests.

**Estimated scope:** Medium slices

## Task F7: Implement reporting periods and deterministic metric queries

**Description:** Build calendar-period discovery and SQL/Ent query services for catalogue `0.4.0`, preserving session grain, missingness, Saturday–Friday/Jalali/season boundaries, and peer suppression/equal weighting.

**Acceptance criteria:**

- [ ] Every catalogue metric reconciles for day/week/month/season and emits unit, denominator/sample/coverage, dataset snapshot, metric version, and calculation time.
- [ ] `Paid` plus `Verified`, verification completion, retry recovery, timing percentiles, fees, and `NoAttempt` semantics match approved formulas.
- [ ] Peer outputs exclude the active merchant, weight merchants equally, invert lower-is-better percentiles, and suppress below both thresholds.

**Verification:** Independent SQL oracle tests, boundary fixtures, large/skewed merchant fixtures, null/zero/unknown-status tests, and measured query plans.

**Dependencies:** F2 and trusted F4 dataset fixtures.

**Files likely touched:** `internal/analytics/periods/`, `internal/analytics/metrics/`, peer queries, fixtures/tests.

**Estimated scope:** Medium slices by section

## Task F8: Implement Redis/Asynq analytics run orchestration

**Description:** Implement idempotent four-section runs, dataset-versioned 30-day cache keys, Asynq `v0.26.0` jobs, bounded concurrency, retries, rate limits, event history, and sibling-failure isolation.

**Acceptance criteria:**

- [ ] Equivalent starts single-flight; all sections begin in bounded parallel and publish ready results atomically/independently.
- [ ] Attempts time out at 60 seconds and only transient failures receive two exponential-backoff/jitter retries.
- [ ] Redis loss yields safe cold recomputation; one-hour events/15-second heartbeat and independent user/merchant limits match specs.

**Verification:** Queue/cache integration, race, failure, retry timing, Redis-loss, deduplication, replay, rate-limit, and sensitive-payload tests.

**Dependencies:** F1, F6, and F7.

**Files likely touched:** `internal/analytics/runs/`, `internal/analytics/jobs/`, Redis adapters, worker root/tests.

**Estimated scope:** Medium slices

## Task F9: Implement tenant-scoped detail and export providers

**Description:** Build the shared authorized query/projection for unverified-paid, failed, and retried details, deterministic keyset pagination, export preflight/count/stream inputs, and operator audit metadata.

**Acceptance criteria:**

- [ ] Every query requires session-derived merchant scope and approved period/filter allowlists.
- [ ] Detail/export projections reconcile to metrics and exclude payer tokens, peer rows, credentials, and internal keys.
- [ ] Stable ordering, cursor inputs, 100,000-row/50-MiB/60-second export caps, and audit metadata are available to the API adapter.

**Verification:** Tenant/tampering/reconciliation/pagination/snapshot/cap/query-plan/leak tests against PostgreSQL.

**Dependencies:** F2, F6, and F7.

**Files likely touched:** `internal/analytics/details/`, cursor/projection package, export preflight, audit/tests.

**Estimated scope:** Medium slices

## Task F10: Stabilize provider interfaces and backend integration

**Description:** Wire Identity, period, analytics-run/event, detail/export, audit, PostgreSQL, and Redis providers behind narrow interfaces consumed by the Merchant API.

**Acceptance criteria:**

- [ ] Composition roots wire explicit constructors with bounded pools/timeouts and graceful lifecycle.
- [ ] Provider contracts match OpenAPI semantics without importing HTTP types into domain packages.
- [ ] A non-HTTP integration harness completes seed/login/period/run/partial-result/detail/export-provider flows for two isolated merchants.

**Verification:** Full provider integration, race, shutdown, dependency-loss, tenant-boundary, and safe-log tests.

**Dependencies:** F4–F9.

**Files likely touched:** provider interfaces/adapters, `cmd/api/`, `cmd/worker/`, integration tests.

**Estimated scope:** Medium (split wiring and integration)

## Checkpoint F: Backend provider readiness

- [ ] Tasks F1–F10 and every foundation acceptance criterion pass.
- [ ] Full pinned import, two-merchant identity, metric reconciliation, Redis-loss, and tenant isolation evidence is reviewed.
- [ ] Human approves provider readiness before completing HTTP feature slices.

## Task 1: Establish the OpenAPI toolchain and conformance harness

**Description:** Select, approve, and pin the Go OpenAPI generation/validation tooling; make contract validation and generated-artifact drift checks reproducible from a clean checkout.

**Acceptance criteria:**

- [ ] One approved pinned toolchain validates OpenAPI 3.1 and produces only the intended server/types artifacts.
- [ ] Generated artifacts contain no hand-edited business logic and regenerate with a clean Git diff.
- [ ] Stable `operationId` values, schemas, media types, and security requirements are covered by a contract test harness.

**Verification:**

- [ ] Run the documented OpenAPI validate/generate/check commands from a clean checkout.
- [ ] Run focused generator/conformance tests and confirm a deliberate contract mismatch fails.
- [ ] Review generated and lockfile/checksum diffs; approve any new dependency before installation.

**Dependencies:** Explicit implementation authorization and F1.

**Files likely touched:**

- `go.mod`
- `go.sum`
- `internal/api/contract/`
- `internal/api/contract_test.go`
- approved command/tool configuration

**Estimated scope:** Medium (3–5 files/artifact groups)

## Task 2: Build the shared HTTP boundary and protected-request middleware

**Description:** Create the API server boundary for `/api/v1`, central Problem Details mapping, request ids, 16 KiB body/8 KiB target limits, no-store/security headers, session resolution, and CSRF/origin enforcement.

**Acceptance criteria:**

- [ ] All protected handlers receive merchant/user context only from a verified server-side session.
- [ ] Boundary failures use the approved RFC 9457 schema/code/status mapping without internal details.
- [ ] Request limits, unknown fields/parameters, CSRF/origin checks, and no-store/security headers behave as specified.

**Verification:**

- [ ] Focused middleware tests cover missing/expired/revoked sessions, malformed input, size limits, request ids, CSRF, and origin/fetch metadata.
- [ ] Cross-tenant client inputs cannot create or replace merchant context.
- [ ] `go test ./...` and `go vet ./...` pass.

**Dependencies:** Task 1 and F6 provider interface.

**Files likely touched:**

- `internal/api/http/server.go`
- `internal/api/http/problem.go`
- `internal/api/http/middleware.go`
- `internal/api/http/middleware_test.go`

**Estimated scope:** Medium (4 files)

## Checkpoint 1: Boundary readiness

- [ ] Tasks 1–2 acceptance criteria pass.
- [ ] OpenAPI generation is reproducible with a clean diff.
- [ ] Human review confirms the tenant and error boundaries before feature handlers begin.

## Task 3: Implement authentication session endpoints

**Description:** Implement login, current-session, and logout as one vertical slice through the approved Identity provider, including secure cookie and generic-failure behavior.

**Acceptance criteria:**

- [ ] Login creates a revocable session and sets only the approved `__Host-zpa_session` cookie attributes.
- [ ] Current-session JSON matches OpenAPI and exposes a session-bound CSRF token but no authentication token.
- [ ] Logout revokes server state before clearing the cookie; expected credential failures remain indistinguishable and throttled.

**Verification:**

- [ ] Integration tests cover success, enumeration resistance, cookie flags, rotation, expiry, revocation, throttling, CSRF, and dependency failure.
- [ ] OpenAPI conformance tests cover all auth statuses and headers.
- [ ] `go test -race ./...` passes for affected packages.

**Dependencies:** Task 2 and F6.

**Files likely touched:**

- `internal/api/http/auth_handler.go`
- `internal/api/http/auth_handler_test.go`
- `internal/api/identity.go`
- `cmd/api/main.go`

**Estimated scope:** Medium (4 files)

## Task 4: Implement approved reporting-period discovery

**Description:** Return only allowlisted day/week/Jalali-month/Persian-season periods with exact UTC/Tehran boundaries, ready Persian labels, and the approved default.

**Acceptance criteria:**

- [ ] Period responses match OpenAPI and use backend-calculated half-open boundaries.
- [ ] Iranian weeks start Saturday; default selection is the latest complete Jalali month containing source data.
- [ ] Unknown parameters/types are rejected and arbitrary date ranges are impossible.

**Verification:**

- [ ] Table tests cover all period types, month/year boundaries, Saturday–Friday weeks, empty source ranges, and timezone independence.
- [ ] Integration tests reconcile returned ids/boundaries with `calendar_dates`.
- [ ] OpenAPI examples and `go test ./...` pass.

**Dependencies:** Task 2 and F7.

**Files likely touched:**

- `internal/api/http/period_handler.go`
- `internal/api/http/period_handler_test.go`
- `internal/api/periods.go`

**Estimated scope:** Medium (3 files)

## Task 5: Implement analytics-run start and snapshot retrieval

**Description:** Implement idempotent run creation/current-run resolution and tenant-scoped snapshot retrieval for all four sections.

**Acceptance criteria:**

- [ ] Same intent/key replays safely, key/body conflicts fail, and equivalent concurrent starts resolve one current run.
- [ ] Run snapshots serialize every section state/result exactly as OpenAPI defines.
- [ ] Missing and cross-tenant run ids return the same `404 RESOURCE_NOT_FOUND` response.

**Verification:**

- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, the immutable dataset snapshot, and no duplicate jobs.
- [ ] Contract tests cover `200`, `202`, `401`, `403`, `404`, `422`, `429`, and dependency failures.
- [ ] Integration tests assert that rejected/limited requests enqueue no work.

**Dependencies:** Tasks 1–2 and F8.

**Files likely touched:**

- `internal/api/http/run_handler.go`
- `internal/api/http/run_handler_test.go`
- `internal/api/analytics.go`
- `internal/api/idempotency.go`

**Estimated scope:** Medium (4 files)

## Checkpoint 2: Core API flow

- [ ] Tasks 3–5 acceptance criteria pass.
- [ ] Authenticated merchant flow works through the actual HTTP server and test dependencies.
- [ ] Cross-tenant and idempotency adversarial tests pass.
- [ ] Human review approves the core flow before streaming/detail work.

## Task 6: Implement authenticated SSE delivery and recovery

**Description:** Stream typed run/section events with 15-second heartbeats, one-hour authorized replay, snapshot reset, and session-aware disconnect/reconnect behavior.

**Acceptance criteria:**

- [ ] Event framing, ids, names, and JSON data match the approved SSE schemas and UTF-8 media type.
- [ ] Retained ids replay only later authorized events; unavailable ids reset with `run.snapshot` and never disclose another tenant.
- [ ] Session expiry/revocation, connection limits, and completed-run closure behave as specified.

**Verification:**

- [ ] Integration tests cover ordered partial readiness, failure isolation, suppression, heartbeat timing, replay, reset, disconnect, revocation, and user/merchant connection limits.
- [ ] Proxy-style tests assert no buffering/compression and required stream headers.
- [ ] Race tests cover concurrent publish/replay/disconnect paths.

**Dependencies:** Task 5 and F8.

**Files likely touched:**

- `internal/api/http/sse_handler.go`
- `internal/api/http/sse_handler_test.go`
- `internal/api/events.go`
- `internal/api/events_test.go`

**Estimated scope:** Medium (4 files)

## Task 7: Implement exhausted-section manual retry

**Description:** Add the idempotent retry-attempt subresource for one exhausted failed section without disturbing ready siblings.

**Acceptance criteria:**

- [ ] Only an exhausted, retryable failed section for the current run/dataset snapshot is accepted.
- [ ] Retry idempotency and independent user/merchant throttles prevent duplicate work.
- [ ] Ready siblings remain unchanged and stale/cross-tenant requests fail safely.

**Verification:**

- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale dataset snapshots, throttling, cross-tenant ids, and no sibling restart.
- [ ] Rejected retries enqueue no work.
- [ ] OpenAPI conformance and `go test -race ./...` pass.

**Dependencies:** Tasks 5–6.

**Files likely touched:**

- `internal/api/http/retry_handler.go`
- `internal/api/http/retry_handler_test.go`
- `internal/api/retry.go`

**Estimated scope:** Medium (3 files)

## Checkpoint 3: Asynchronous behavior

- [ ] Tasks 6–7 acceptance criteria pass.
- [ ] SSE and retry runtime checks pass through the actual queue/cache test boundary.
- [ ] Human review confirms recovery and rate-limit behavior.

## Task 8: Implement cursor-paginated session detail views

**Description:** Implement the three approved detail views with deterministic ordering, opaque bound cursors, default 25/maximum 100 rows, and a strict field allowlist.

**Acceptance criteria:**

- [ ] `UNVERIFIED_PAID`, `FAILED`, and `RETRIED` filters match catalogue definitions and reconcile to aggregates.
- [ ] Cursors bind tenant, canonical filters, ordering, expiry, and last row; invalid reuse returns generic `INVALID_CURSOR`.
- [ ] Detail rows expose only approved fields and the tenant-scoped opaque terminal reference.

**Verification:**

- [ ] Tests cover page boundaries, duplicate timestamps, cursor tampering/expiry/filter change/cross-tenant reuse, limits, and empty results.
- [ ] Integration queries reconcile representative merchants and approved periods.
- [ ] Sensitive-field absence and measured query-plan checks pass.

**Dependencies:** Tasks 2 and 4 plus F9.

**Files likely touched:**

- `internal/api/http/detail_handler.go`
- `internal/api/http/detail_handler_test.go`
- `internal/api/details.go`
- `internal/api/cursor.go`
- `internal/api/cursor_test.go`

**Estimated scope:** Medium (5 files)

## Task 9: Implement safeguarded Persian CSV export

**Description:** Stream the same authorized detail predicate/projection as Persian CSV with BOM, Persian spaced headers/statuses, machine-safe numeric/ISO values, formula neutralization, and approved caps.

**Acceptance criteria:**

- [ ] CSV and JSON share one canonical tenant/filter/projection policy; CSV adds no unavailable field.
- [ ] Output matches the approved encoding, headers, labels, quoting, disposition, no-store, and `nosniff` contract.
- [ ] Preflight rejects over 100,000 rows/50 MiB, generation ends within 60 seconds, throttling applies, and interrupted streams are audited as incomplete.

**Verification:**

- [ ] Golden tests cover Persian output, BOM, CRLF, quotes/newlines, all status mappings, and stable headers with spaces.
- [ ] Security tests cover `=`, `+`, `-`, `@`, tab, CR, LF formula prefixes and forbidden-field absence.
- [ ] Integration tests cover filter parity, snapshot consistency, caps, deadline, disconnect, throttle, and content headers.

**Dependencies:** Task 8.

**Files likely touched:**

- `internal/api/http/export_handler.go`
- `internal/api/http/export_handler_test.go`
- `internal/api/export.go`
- `internal/api/export_test.go`

**Estimated scope:** Medium (4 files)

## Checkpoint 4: Reconciliation and data boundary

- [ ] Tasks 8–9 acceptance criteria pass.
- [ ] Detail/CSV reconciliation and sensitive-data negative tests pass.
- [ ] Export resource limits hold under representative merchant volume.
- [ ] Human review approves the merchant evidence boundary.

## Task 10: Publish the stable API problem-type catalogue

**Description:** Document every stable application error code and make each approved `https://zarinpal.amiirkhl.ir/docs/errors/{error-slug}` URI resolve to safe human-readable guidance.

**Acceptance criteria:**

- [ ] Every OpenAPI problem code maps to one stable slug, title, status, meaning, safe resolution guidance, and retry rule.
- [ ] Published pages expose no internal topology, SQL, stack, secret, tenant existence, or security-control detail useful for abuse.
- [ ] Contract tests fail when a returned problem type/code lacks catalogue documentation.

**Verification:**

- [ ] Link checks confirm every production type URI resolves successfully.
- [ ] OpenAPI and catalogue mapping tests pass with no duplicate slug/code.
- [ ] Security review approves public wording.

**Dependencies:** Tasks 2–9 for the final error-code set; hosting mechanism approval.

**Files likely touched:**

- `docs/ERROR-CATALOGUE.md`
- error-page source/template location approved by the hosting plan
- `internal/api/http/problem_catalogue_test.go`

**Estimated scope:** Medium (2–3 files/artifact groups)

## Task 11: Add API audit and observability instrumentation

**Description:** Instrument the approved audit events, metrics, logs, and traces with bounded-cardinality safe metadata and 90-day analytics/export-audit retention.

**Acceptance criteria:**

- [ ] Required auth/run/SSE/detail/export audit outcomes and operational metrics are emitted with correlation ids.
- [ ] Only authorized backend operators can access analytics/export audit records; automatic deletion occurs after 90 days.
- [ ] Telemetry excludes credentials, tokens, raw cursors/filters, payment rows, payer tokens, peer data, and high-cardinality sensitive labels.

**Verification:**

- [ ] Tests assert required event/metric emission for success, rejection, failure, timeout, and interrupted export.
- [ ] Negative leak tests inspect structured logs, traces, and audit records.
- [ ] Retention/access-control tests pass using an accelerated test clock.

**Dependencies:** Tasks 3–10 and the approved audit/observability adapters.

**Files likely touched:**

- `internal/api/observability.go`
- `internal/api/audit.go`
- `internal/api/observability_test.go`
- `internal/api/audit_test.go`

**Estimated scope:** Medium (4 files)

## Task 12: Complete contract, security, race, and performance verification

**Description:** Run and close the full implementation Definition of Done across all API operations and critical failure/abuse paths.

**Acceptance criteria:**

- [ ] Every OpenAPI operation/status/media type/header/security requirement has automated contract coverage.
- [ ] Tenant isolation, session/CSRF, idempotency, SSE, cursor, export, audit, and sensitive-field abuse suites pass.
- [ ] Representative detail/export query plans and resource bounds meet approved behavior with no unreviewed exception.

**Verification:**

- [ ] Run `go test ./...`, `go test -race ./...`, `go vet ./...`, `golangci-lint run`, and `govulncheck ./...`.
- [ ] Run reproducible OpenAPI validation/generation and confirm a clean Git diff.
- [ ] Exercise critical merchant flows end to end through the deployed-style proxy/test environment.

**Dependencies:** Tasks 1–11.

**Files likely touched:**

- API contract/integration test packages
- approved test fixtures containing no sensitive source data
- API runbook/documentation corrections if verification finds drift

**Estimated scope:** Medium (3–5 files/artifact groups)

## Checkpoint 5: Complete

- [ ] Tasks 1–12 and every checkpoint pass.
- [ ] Full project Definition of Done is satisfied.
- [ ] No unrelated frontend, deployment, physical-schema, ERD, or raw-data changes are included.
- [ ] Human review approves completion before merge or deployment.