docs(plan): expand backend implementation roadmap

tasks/todo.md

@@ -1,10 +1,196 @@
-# Merchant Analytics API Task List
+# ZarinPal Merchant Analytics Backend Task List
 
-- Status: Approved task breakdown — implementation not authorized
+- Status: Expanded task breakdown ready for stakeholder review — implementation not authorized
 - Plan: `tasks/plan.md`
 - Contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0`
 
-Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files.
+Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. Foundation tasks use `F` identifiers; the previously approved API tasks retain their stable numeric identifiers.
+
+## Task F1: Establish the Go workspace and quality toolchain
+
+**Description:** Create the thin command-root/modular-monolith skeleton, validated configuration boundary, structured logging baseline, and pinned repository quality commands without implementing product behavior.
+
+**Acceptance criteria:**
+
+- [ ] `cmd/api`, `cmd/worker`, `cmd/importer`, and migration/admin composition roots are thin and business packages live under `internal/`.
+- [ ] Manual constructor wiring, context propagation, validated environment configuration, and safe structured logs follow approved ADRs/AGENTS.
+- [ ] Pinned build/test/race/vet/lint/vulnerability/generation commands are documented and reproducible.
+
+**Verification:** Run every scaffold quality command from a clean checkout; confirm no service starts implicitly and dependency/checksum diffs are approved.
+
+**Dependencies:** Explicit implementation authorization.
+
+**Files likely touched:** `go.mod`, `go.sum`, `cmd/`, `internal/platform/`, quality-tool configuration.
+
+**Estimated scope:** Medium (split command roots if more than five files)
+
+## Task F2: Implement the normalized Ent model and migration baseline
+
+**Description:** Translate the approved physical schema into Ent entities/relationships and reviewed Atlas SQL migrations, including identity, import, payment, calendar, and audit tables without cache-revision triggers.
+
+**Acceptance criteria:**
+
+- [ ] Tables, FKs, checks, unique/partial indexes, nullability, restrictive deletion, and 30/90-day audit retention match the physical schema.
+- [ ] `payment_sessions.import_run_id` is required; `NoAttempt` never becomes an attempt; terminal verification method and session-grain money are stored once.
+- [ ] Clean creation and supported upgrade migrations are versioned, reviewed, and never run from API startup.
+
+**Verification:** Run zero-to-head/upgrade migration tests, schema introspection/ERD comparison, constraint negatives, and representative query-plan checks.
+
+**Dependencies:** F1.
+
+**Files likely touched:** `internal/data/schema/`, `ent/schema/`, `migrations/`, migration tests, Atlas configuration.
+
+**Estimated scope:** Medium slices by schema group
+
+## Task F3: Implement pinned archive preflight and bounded staging
+
+**Description:** Build the one-shot importer boundary that validates the fixed file/checksum/schema/resource caps and streams strict CSV/gzip records through pgx `COPY` into run-scoped staging.
+
+**Acceptance criteria:**
+
+- [ ] Fixed filename, SHA-256, 22-column schema, UTF-8/gzip integrity, byte/row/record caps, and regular read-only path checks fail closed.
+- [ ] Parsing is bounded-memory, independent of file/session order, and records safe row numbers without logging source values.
+- [ ] One PostgreSQL advisory lock prevents concurrent validate/import/recovery commands.
+
+**Verification:** Unit/fuzz tests cover malformed CSV/UTF-8/gzip, overflow, traversal/symlink, expansion, cancellation, caps, and redaction; staging load stays within measured memory.
+
+**Dependencies:** F1–F2.
+
+**Files likely touched:** `cmd/importer/`, `internal/importing/preflight/`, `internal/importing/staging/`, focused tests.
+
+**Estimated scope:** Medium, split preflight and staging commits
+
+## Task F4: Implement validation, atomic promotion, reports, and recovery
+
+**Description:** Add set-based relationship validation, warning/fatal classification, exact reconciliation, transactional normalization, redacted reports, staging cleanup, no-op detection, and restart-from-source recovery.
+
+**Acceptance criteria:**
+
+- [ ] Any fatal issue promotes zero normalized facts; documented timestamp/sequence/null anomalies remain unchanged warnings.
+- [ ] Pinned counts and per-merchant/status reconciliations pass before the sole import run becomes completed.
+- [ ] Success/failure removes raw staging; interrupted recovery cleans only its run scope and restarts from the immutable archive.
+
+**Verification:** Full-archive rehearsal plus relationship/rollback/fault-injection/concurrency/no-op/recovery/leak tests; JSON and Markdown reports reconcile.
+
+**Dependencies:** F3.
+
+**Files likely touched:** `internal/importing/validation/`, `internal/importing/promotion/`, `internal/importing/report/`, recovery command/tests.
+
+**Estimated scope:** Medium slices; never one XL commit
+
+## Task F5: Implement synthetic merchant profiles and demo-account seeding
+
+**Description:** Generate deterministic non-sensitive merchant display profiles and exactly one initial demo user per merchant through a separate secure administrative flow.
+
+**Acceptance criteria:**
+
+- [ ] Display names are deterministic, clearly synthetic in storage, and stable across identical imports.
+- [ ] Usernames are unique and credentials use approved Argon2id hashing; plaintext is emitted once only to an explicit protected output.
+- [ ] Re-running the seed is idempotent and never derives passwords from merchant/source keys.
+
+**Verification:** Determinism/idempotency/hash/permission/redaction tests and two-merchant credential smoke fixture.
+
+**Dependencies:** F2 and completed F4 provider behavior.
+
+**Files likely touched:** `cmd/admin/`, `internal/identity/seeding/`, profile generator/tests, operator output adapter.
+
+**Estimated scope:** Medium (3–5 files)
+
+## Task F6: Implement password authentication and server-side sessions
+
+**Description:** Implement demo credential verification, generic throttled outcomes, two-active-session enforcement, opaque token digests, CSRF binding, expiry/activity coalescing, logout/revocation, and authentication audit retention.
+
+**Acceptance criteria:**
+
+- [ ] Authentication never reveals account existence and returns tenant context only from a valid server-side session.
+- [ ] Cookie/token/CSRF, idle/absolute expiry, revocation, disabled account, and two-session behavior match Identity spec.
+- [ ] Raw credentials/tokens/IP/user-agent values are absent from storage, logs, and audit evidence.
+
+**Verification:** Table/race/integration tests cover enumeration, throttles, fixation, rotation, expiry, revocation, concurrency, and 30-day cleanup.
+
+**Dependencies:** F2 and F5.
+
+**Files likely touched:** `internal/identity/`, session repository, throttling/audit adapters, focused tests.
+
+**Estimated scope:** Medium slices
+
+## Task F7: Implement reporting periods and deterministic metric queries
+
+**Description:** Build calendar-period discovery and SQL/Ent query services for catalogue `0.4.0`, preserving session grain, missingness, Saturday–Friday/Jalali/season boundaries, and peer suppression/equal weighting.
+
+**Acceptance criteria:**
+
+- [ ] Every catalogue metric reconciles for day/week/month/season and emits unit, denominator/sample/coverage, dataset snapshot, metric version, and calculation time.
+- [ ] `Paid` plus `Verified`, verification completion, retry recovery, timing percentiles, fees, and `NoAttempt` semantics match approved formulas.
+- [ ] Peer outputs exclude the active merchant, weight merchants equally, invert lower-is-better percentiles, and suppress below both thresholds.
+
+**Verification:** Independent SQL oracle tests, boundary fixtures, large/skewed merchant fixtures, null/zero/unknown-status tests, and measured query plans.
+
+**Dependencies:** F2 and trusted F4 dataset fixtures.
+
+**Files likely touched:** `internal/analytics/periods/`, `internal/analytics/metrics/`, peer queries, fixtures/tests.
+
+**Estimated scope:** Medium slices by section
+
+## Task F8: Implement Redis/Asynq analytics run orchestration
+
+**Description:** Implement idempotent four-section runs, dataset-versioned 30-day cache keys, Asynq `v0.26.0` jobs, bounded concurrency, retries, rate limits, event history, and sibling-failure isolation.
+
+**Acceptance criteria:**
+
+- [ ] Equivalent starts single-flight; all sections begin in bounded parallel and publish ready results atomically/independently.
+- [ ] Attempts time out at 60 seconds and only transient failures receive two exponential-backoff/jitter retries.
+- [ ] Redis loss yields safe cold recomputation; one-hour events/15-second heartbeat and independent user/merchant limits match specs.
+
+**Verification:** Queue/cache integration, race, failure, retry timing, Redis-loss, deduplication, replay, rate-limit, and sensitive-payload tests.
+
+**Dependencies:** F1, F6, and F7.
+
+**Files likely touched:** `internal/analytics/runs/`, `internal/analytics/jobs/`, Redis adapters, worker root/tests.
+
+**Estimated scope:** Medium slices
+
+## Task F9: Implement tenant-scoped detail and export providers
+
+**Description:** Build the shared authorized query/projection for unverified-paid, failed, and retried details, deterministic keyset pagination, export preflight/count/stream inputs, and operator audit metadata.
+
+**Acceptance criteria:**
+
+- [ ] Every query requires session-derived merchant scope and approved period/filter allowlists.
+- [ ] Detail/export projections reconcile to metrics and exclude payer tokens, peer rows, credentials, and internal keys.
+- [ ] Stable ordering, cursor inputs, 100,000-row/50-MiB/60-second export caps, and audit metadata are available to the API adapter.
+
+**Verification:** Tenant/tampering/reconciliation/pagination/snapshot/cap/query-plan/leak tests against PostgreSQL.
+
+**Dependencies:** F2, F6, and F7.
+
+**Files likely touched:** `internal/analytics/details/`, cursor/projection package, export preflight, audit/tests.
+
+**Estimated scope:** Medium slices
+
+## Task F10: Stabilize provider interfaces and backend integration
+
+**Description:** Wire Identity, period, analytics-run/event, detail/export, audit, PostgreSQL, and Redis providers behind narrow interfaces consumed by the Merchant API.
+
+**Acceptance criteria:**
+
+- [ ] Composition roots wire explicit constructors with bounded pools/timeouts and graceful lifecycle.
+- [ ] Provider contracts match OpenAPI semantics without importing HTTP types into domain packages.
+- [ ] A non-HTTP integration harness completes seed/login/period/run/partial-result/detail/export-provider flows for two isolated merchants.
+
+**Verification:** Full provider integration, race, shutdown, dependency-loss, tenant-boundary, and safe-log tests.
+
+**Dependencies:** F4–F9.
+
+**Files likely touched:** provider interfaces/adapters, `cmd/api/`, `cmd/worker/`, integration tests.
+
+**Estimated scope:** Medium (split wiring and integration)
+
+## Checkpoint F: Backend provider readiness
+
+- [ ] Tasks F1–F10 and every foundation acceptance criterion pass.
+- [ ] Full pinned import, two-merchant identity, metric reconciliation, Redis-loss, and tenant isolation evidence is reviewed.
+- [ ] Human approves provider readiness before completing HTTP feature slices.
 
 ## Task 1: Establish the OpenAPI toolchain and conformance harness
 
@@ -22,7 +208,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 - [ ] Run focused generator/conformance tests and confirm a deliberate contract mismatch fails.
 - [ ] Review generated and lockfile/checksum diffs; approve any new dependency before installation.
 
-**Dependencies:** Approved implementation authorization and separately approved Go scaffold.
+**Dependencies:** Explicit implementation authorization and F1.
 
 **Files likely touched:**
 
@@ -50,7 +236,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 - [ ] Cross-tenant client inputs cannot create or replace merchant context.
 - [ ] `go test ./...` and `go vet ./...` pass.
 
-**Dependencies:** Task 1 and approved Identity session-verification interface.
+**Dependencies:** Task 1 and F6 provider interface.
 
 **Files likely touched:**
 
@@ -83,7 +269,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 - [ ] OpenAPI conformance tests cover all auth statuses and headers.
 - [ ] `go test -race ./...` passes for affected packages.
 
-**Dependencies:** Task 2 and approved Identity provider implementation.
+**Dependencies:** Task 2 and F6.
 
 **Files likely touched:**
 
@@ -110,7 +296,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 - [ ] Integration tests reconcile returned ids/boundaries with `calendar_dates`.
 - [ ] OpenAPI examples and `go test ./...` pass.
 
-**Dependencies:** Task 2 and approved calendar/period provider implementation.
+**Dependencies:** Task 2 and F7.
 
 **Files likely touched:**
 
@@ -132,11 +318,11 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 
 **Verification:**
 
-- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, current revisions, and no duplicate jobs.
+- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, the immutable dataset snapshot, and no duplicate jobs.
 - [ ] Contract tests cover `200`, `202`, `401`, `403`, `404`, `422`, `429`, and dependency failures.
 - [ ] Integration tests assert that rejected/limited requests enqueue no work.
 
-**Dependencies:** Tasks 1–2 and approved Analytics run provider implementation.
+**Dependencies:** Tasks 1–2 and F8.
 
 **Files likely touched:**
 
@@ -170,7 +356,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 - [ ] Proxy-style tests assert no buffering/compression and required stream headers.
 - [ ] Race tests cover concurrent publish/replay/disconnect paths.
 
-**Dependencies:** Task 5 and approved Analytics event provider implementation.
+**Dependencies:** Task 5 and F8.
 
 **Files likely touched:**
 
@@ -187,13 +373,13 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 
 **Acceptance criteria:**
 
-- [ ] Only an exhausted, retryable failed section for the current run/revisions is accepted.
+- [ ] Only an exhausted, retryable failed section for the current run/dataset snapshot is accepted.
 - [ ] Retry idempotency and independent user/merchant throttles prevent duplicate work.
 - [ ] Ready siblings remain unchanged and stale/cross-tenant requests fail safely.
 
 **Verification:**
 
-- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale revisions, throttling, cross-tenant ids, and no sibling restart.
+- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale dataset snapshots, throttling, cross-tenant ids, and no sibling restart.
 - [ ] Rejected retries enqueue no work.
 - [ ] OpenAPI conformance and `go test -race ./...` pass.
 
@@ -229,7 +415,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
 - [ ] Integration queries reconcile representative merchants and approved periods.
 - [ ] Sensitive-field absence and measured query-plan checks pass.
 
-**Dependencies:** Tasks 2 and 4 plus approved Analytics detail provider implementation.
+**Dependencies:** Tasks 2 and 4 plus F9.
 
 **Files likely touched:**