diff --git a/tasks/plan.md b/tasks/plan.md index 1262b53..d971cd3 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -1,18 +1,18 @@ -# Implementation Plan: Merchant Analytics API +# Implementation Plan: ZarinPal Merchant Analytics Backend -- Status: Approved plan and task breakdown — implementation not authorized -- Capability: Merchant API -- Approved contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0` +- Status: Expanded implementation plan ready for stakeholder review — implementation not authorized +- Capabilities: Data foundation, demo identity, analytics engine, and Merchant API +- Approved contracts: physical schema, import workflow/validation, Identity, Analytics, metric catalogue `0.4.0`, API, and OpenAPI `1.0.0` - Task list: `tasks/todo.md` - Last updated: 2026-08-20 ## Overview -Implement the approved `/api/v1` REST, SSE, paginated-detail, and CSV interfaces as the HTTP adapter of the capability-oriented Go modular monolith. The API layer validates untrusted HTTP input, resolves the merchant from the server-side session, calls approved Identity/Analytics provider contracts, and serializes only OpenAPI-defined output. It does not own metric formulas, normalized payment facts, queue behavior, or deployment configuration. +Build the complete Go capability-oriented modular monolith needed by the dashboard: reviewed PostgreSQL migrations and normalized entities, the guarded one-time importer, demo merchant identity, deterministic analytics queries, Redis/Asynq orchestration, and the approved `/api/v1` REST/SSE/detail/CSV adapter. Deployment configuration remains owned by the deployment repository. ## Preconditions and boundaries -- The approved Identity, Data Foundation, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative. +- The approved Identity, Data Foundation, physical schema, importer, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative. - Architecture and dependency injection are already approved: capability-oriented modular monolith with manual constructor wiring in thin `cmd/` composition roots. - This plan does not authorize `go.mod`, packages, generated files, dependencies, migrations, containers, or executable code. - Implementation starts only after the plan and task list are approved and the user separately authorizes implementation. @@ -32,38 +32,67 @@ Implement the approved `/api/v1` REST, SSE, paginated-detail, and CSV interfaces ## Dependency graph ```text -Approved OpenAPI 1.0.0 + backend scaffold - | - v -Contract toolchain and HTTP boundary - | - +---------+----------+ - | | - v v -Identity adapter Analytics provider adapters - | | - v +---------+---------+ -Auth endpoints Periods Runs/snapshot - | - +------+------+ - v v - SSE Section retry - -Analytics provider adapters - | - v - Detail JSON pagination - | - v - CSV export - | - v - Audit, observability, and full conformance +Go scaffold/tooling + | + v +Ent model + reviewed Atlas migrations + | + +--> pgx staging importer --> validation/promotion/recovery + | | + | v + | sole immutable dataset snapshot + | + +--> demo identity/session service + | + +--> calendar + metric query kernel + | + v + Redis/Asynq run orchestration + | + v +Approved OpenAPI --> HTTP boundary --> auth/period/run/SSE/detail/CSV + | + v + audit + full verification ``` ## Ordered task index -### Phase 1: Contract and HTTP boundary +### Foundation phase A: repository and schema + +- [ ] Task F1: Establish the Go workspace, command roots, configuration, and quality toolchain. +- [ ] Task F2: Implement the normalized Ent model and reviewed Atlas migration baseline. + +### Checkpoint F-A: Schema foundation + +- [ ] Clean migrations create the approved physical schema and ERD relationships without triggers or revision tables. +- [ ] Build, unit-test, lint, vulnerability, and migration commands are reproducible. + +### Foundation phase B: one-time dataset bootstrap + +- [ ] Task F3: Implement pinned archive preflight and bounded pgx staging. +- [ ] Task F4: Implement set-based validation, atomic promotion, redacted reports, and recovery. +- [ ] Task F5: Implement deterministic synthetic merchant profiles and secure demo-account seeding. + +### Checkpoint F-B: Trusted dataset + +- [ ] The full pinned archive reconciles to approved counts and the second identical run is a no-op. +- [ ] Fatal input, interruption, concurrency, and sensitive-data leak tests pass. + +### Foundation phase C: identity and analytics providers + +- [ ] Task F6: Implement password authentication and revocable server-side merchant sessions. +- [ ] Task F7: Implement calendar periods and the deterministic metric query kernel. +- [ ] Task F8: Implement Redis/Asynq analytics runs, section cache, retries, and event history. +- [ ] Task F9: Implement tenant-scoped detail queries and shared export projection. +- [ ] Task F10: Stabilize provider interfaces and complete backend-provider integration checks. + +### Checkpoint F-C: Provider readiness + +- [ ] Identity, periods, metrics, peer suppression, runs, events, detail, and export providers pass PostgreSQL/Redis integration tests. +- [ ] Provider contracts expose the immutable dataset snapshot and no client-supplied merchant authority. + +### API phase 1: Contract and HTTP boundary - [ ] Task 1: Establish the OpenAPI toolchain and conformance harness. - [ ] Task 2: Build the shared HTTP boundary and protected-request middleware. @@ -74,7 +103,7 @@ Analytics provider adapters - [ ] Shared Problem Details, request limits, request ids, cookie authentication, and CSRF/origin behavior pass focused tests. - [ ] Human review confirms no application handler can obtain tenant authority from client input. -### Phase 2: Core vertical slices +### API phase 2: Core vertical slices - [ ] Task 3: Implement login, current-session, and logout endpoints. - [ ] Task 4: Implement approved reporting-period discovery. @@ -86,7 +115,7 @@ Analytics provider adapters - [ ] Cross-tenant ids are indistinguishable from missing ids. - [ ] OpenAPI examples and focused integration tests pass. -### Phase 3: Partial delivery and recovery +### API phase 3: Partial delivery and recovery - [ ] Task 6: Implement authenticated SSE delivery and recovery. - [ ] Task 7: Implement exhausted-section manual retry. @@ -96,7 +125,7 @@ Analytics provider adapters - [ ] Independent section completion, heartbeat, replay, snapshot reset, revocation, and sibling-failure isolation pass integration tests. - [ ] Existing start/retry/SSE limits reject before queue or stream work. -### Phase 4: Evidence and export +### API phase 4: Evidence and export - [ ] Task 8: Implement cursor-paginated session detail views. - [ ] Task 9: Implement safeguarded Persian CSV export. @@ -107,7 +136,7 @@ Analytics provider adapters - [ ] JSON and CSV use identical tenant/filter predicates and approved field projections. - [ ] Payer tokens, credentials, peer rows, and internal keys are absent from responses, exports, logs, and traces. -### Phase 5: Operational hardening +### API phase 5: Operational hardening - [ ] Task 10: Publish the stable API problem-type catalogue. - [ ] Task 11: Add API audit and observability instrumentation. @@ -127,6 +156,10 @@ Analytics provider adapters - Write tests that fail before the behavior exists and pass after it is implemented. - Run focused package tests for the changed slice. +- For schema/import slices, compare migrations to the approved ERD, exercise rollback/recovery, and reconcile pinned archive counts without committing source data. +- For metric slices, compare results to an independent SQL oracle across Tehran/Jalali boundaries, missing values, status semantics, peers, and large/skewed merchants. +- For identity slices, exercise enumeration resistance, two-session enforcement, expiry/revocation, CSRF, throttling, tenant isolation, and sensitive-value absence. +- For Redis/Asynq slices, exercise single-flight, independent partial completion, bounded retries, replay, rate limits, and safe cold recomputation after Redis loss. - Validate `docs/openapi.yaml` and regenerate/check generated artifacts without diff. - Run `go test ./...` and `go vet ./...` once the Go scaffold exists. - Inspect the staged diff and scan it for secrets and forbidden data fields before every commit. @@ -140,6 +173,10 @@ Analytics provider adapters ## Parallelization +- F2 follows F1. After F2, importer preflight/staging (F3), metric-query work (F7), and the non-credential portion of synthetic profile design (F5) may proceed independently. +- F4 follows F3. Credential seeding completes only after F4, and session work (F6) follows F5. +- Run orchestration (F8) and detail/export providers (F9) may proceed in parallel after their identity and metric dependencies are stable; F10 integrates them sequentially. +- API contract tooling (Task 1) may proceed alongside provider work after F1, but protected endpoints wait for their named F-provider dependencies. - Tasks 3 and 4 may proceed in parallel only after Tasks 1–2 stabilize shared HTTP types and middleware. - Task 6 and the read-only parts of Task 8 may proceed in parallel after Task 5 and the Analytics provider interfaces stabilize. - Task 9 is sequential after Task 8 because CSV must reuse its authorization, filters, and projection. @@ -150,6 +187,9 @@ Analytics provider adapters | Risk | Impact | Mitigation | |---|---|---| +| Import accepts corrupt/oversized data or partially promotes | Critical | Pinned preflight, bounded streaming, set-based validation, atomic promotion, exact reconciliation, fault-injection and recovery tests | +| Physical schema or metrics drift from approved semantics | Critical | ERD/migration comparison, independent SQL oracle, boundary fixtures, catalogue-version evidence | +| Demo authentication consumes effort or becomes production identity | Medium | Keep the bounded demo-only session scope and preserve the explicit future ZarinPal SSO replacement boundary | | Generated types drift from OpenAPI | High | Reproducible pinned generation plus clean-diff and contract-test gates | | Tenant scope leaks through ids/cursors/replay | Critical | Session-derived merchant context, ownership checks in providers, identical not-found policy, adversarial integration tests | | SSE behavior differs across proxy/browser boundaries | High | Test event framing, heartbeat, buffering headers, replay, reset, disconnect, and revocation through the deployed-style proxy path | diff --git a/tasks/todo.md b/tasks/todo.md index a1461ce..04032e1 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -1,10 +1,196 @@ -# Merchant Analytics API Task List +# ZarinPal Merchant Analytics Backend Task List -- Status: Approved task breakdown — implementation not authorized +- Status: Expanded task breakdown ready for stakeholder review — implementation not authorized - Plan: `tasks/plan.md` - Contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0` -Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. +Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. Foundation tasks use `F` identifiers; the previously approved API tasks retain their stable numeric identifiers. + +## Task F1: Establish the Go workspace and quality toolchain + +**Description:** Create the thin command-root/modular-monolith skeleton, validated configuration boundary, structured logging baseline, and pinned repository quality commands without implementing product behavior. + +**Acceptance criteria:** + +- [ ] `cmd/api`, `cmd/worker`, `cmd/importer`, and migration/admin composition roots are thin and business packages live under `internal/`. +- [ ] Manual constructor wiring, context propagation, validated environment configuration, and safe structured logs follow approved ADRs/AGENTS. +- [ ] Pinned build/test/race/vet/lint/vulnerability/generation commands are documented and reproducible. + +**Verification:** Run every scaffold quality command from a clean checkout; confirm no service starts implicitly and dependency/checksum diffs are approved. + +**Dependencies:** Explicit implementation authorization. + +**Files likely touched:** `go.mod`, `go.sum`, `cmd/`, `internal/platform/`, quality-tool configuration. + +**Estimated scope:** Medium (split command roots if more than five files) + +## Task F2: Implement the normalized Ent model and migration baseline + +**Description:** Translate the approved physical schema into Ent entities/relationships and reviewed Atlas SQL migrations, including identity, import, payment, calendar, and audit tables without cache-revision triggers. + +**Acceptance criteria:** + +- [ ] Tables, FKs, checks, unique/partial indexes, nullability, restrictive deletion, and 30/90-day audit retention match the physical schema. +- [ ] `payment_sessions.import_run_id` is required; `NoAttempt` never becomes an attempt; terminal verification method and session-grain money are stored once. +- [ ] Clean creation and supported upgrade migrations are versioned, reviewed, and never run from API startup. + +**Verification:** Run zero-to-head/upgrade migration tests, schema introspection/ERD comparison, constraint negatives, and representative query-plan checks. + +**Dependencies:** F1. + +**Files likely touched:** `internal/data/schema/`, `ent/schema/`, `migrations/`, migration tests, Atlas configuration. + +**Estimated scope:** Medium slices by schema group + +## Task F3: Implement pinned archive preflight and bounded staging + +**Description:** Build the one-shot importer boundary that validates the fixed file/checksum/schema/resource caps and streams strict CSV/gzip records through pgx `COPY` into run-scoped staging. + +**Acceptance criteria:** + +- [ ] Fixed filename, SHA-256, 22-column schema, UTF-8/gzip integrity, byte/row/record caps, and regular read-only path checks fail closed. +- [ ] Parsing is bounded-memory, independent of file/session order, and records safe row numbers without logging source values. +- [ ] One PostgreSQL advisory lock prevents concurrent validate/import/recovery commands. + +**Verification:** Unit/fuzz tests cover malformed CSV/UTF-8/gzip, overflow, traversal/symlink, expansion, cancellation, caps, and redaction; staging load stays within measured memory. + +**Dependencies:** F1–F2. + +**Files likely touched:** `cmd/importer/`, `internal/importing/preflight/`, `internal/importing/staging/`, focused tests. + +**Estimated scope:** Medium, split preflight and staging commits + +## Task F4: Implement validation, atomic promotion, reports, and recovery + +**Description:** Add set-based relationship validation, warning/fatal classification, exact reconciliation, transactional normalization, redacted reports, staging cleanup, no-op detection, and restart-from-source recovery. + +**Acceptance criteria:** + +- [ ] Any fatal issue promotes zero normalized facts; documented timestamp/sequence/null anomalies remain unchanged warnings. +- [ ] Pinned counts and per-merchant/status reconciliations pass before the sole import run becomes completed. +- [ ] Success/failure removes raw staging; interrupted recovery cleans only its run scope and restarts from the immutable archive. + +**Verification:** Full-archive rehearsal plus relationship/rollback/fault-injection/concurrency/no-op/recovery/leak tests; JSON and Markdown reports reconcile. + +**Dependencies:** F3. + +**Files likely touched:** `internal/importing/validation/`, `internal/importing/promotion/`, `internal/importing/report/`, recovery command/tests. + +**Estimated scope:** Medium slices; never one XL commit + +## Task F5: Implement synthetic merchant profiles and demo-account seeding + +**Description:** Generate deterministic non-sensitive merchant display profiles and exactly one initial demo user per merchant through a separate secure administrative flow. + +**Acceptance criteria:** + +- [ ] Display names are deterministic, clearly synthetic in storage, and stable across identical imports. +- [ ] Usernames are unique and credentials use approved Argon2id hashing; plaintext is emitted once only to an explicit protected output. +- [ ] Re-running the seed is idempotent and never derives passwords from merchant/source keys. + +**Verification:** Determinism/idempotency/hash/permission/redaction tests and two-merchant credential smoke fixture. + +**Dependencies:** F2 and completed F4 provider behavior. + +**Files likely touched:** `cmd/admin/`, `internal/identity/seeding/`, profile generator/tests, operator output adapter. + +**Estimated scope:** Medium (3–5 files) + +## Task F6: Implement password authentication and server-side sessions + +**Description:** Implement demo credential verification, generic throttled outcomes, two-active-session enforcement, opaque token digests, CSRF binding, expiry/activity coalescing, logout/revocation, and authentication audit retention. + +**Acceptance criteria:** + +- [ ] Authentication never reveals account existence and returns tenant context only from a valid server-side session. +- [ ] Cookie/token/CSRF, idle/absolute expiry, revocation, disabled account, and two-session behavior match Identity spec. +- [ ] Raw credentials/tokens/IP/user-agent values are absent from storage, logs, and audit evidence. + +**Verification:** Table/race/integration tests cover enumeration, throttles, fixation, rotation, expiry, revocation, concurrency, and 30-day cleanup. + +**Dependencies:** F2 and F5. + +**Files likely touched:** `internal/identity/`, session repository, throttling/audit adapters, focused tests. + +**Estimated scope:** Medium slices + +## Task F7: Implement reporting periods and deterministic metric queries + +**Description:** Build calendar-period discovery and SQL/Ent query services for catalogue `0.4.0`, preserving session grain, missingness, Saturday–Friday/Jalali/season boundaries, and peer suppression/equal weighting. + +**Acceptance criteria:** + +- [ ] Every catalogue metric reconciles for day/week/month/season and emits unit, denominator/sample/coverage, dataset snapshot, metric version, and calculation time. +- [ ] `Paid` plus `Verified`, verification completion, retry recovery, timing percentiles, fees, and `NoAttempt` semantics match approved formulas. +- [ ] Peer outputs exclude the active merchant, weight merchants equally, invert lower-is-better percentiles, and suppress below both thresholds. + +**Verification:** Independent SQL oracle tests, boundary fixtures, large/skewed merchant fixtures, null/zero/unknown-status tests, and measured query plans. + +**Dependencies:** F2 and trusted F4 dataset fixtures. + +**Files likely touched:** `internal/analytics/periods/`, `internal/analytics/metrics/`, peer queries, fixtures/tests. + +**Estimated scope:** Medium slices by section + +## Task F8: Implement Redis/Asynq analytics run orchestration + +**Description:** Implement idempotent four-section runs, dataset-versioned 30-day cache keys, Asynq `v0.26.0` jobs, bounded concurrency, retries, rate limits, event history, and sibling-failure isolation. + +**Acceptance criteria:** + +- [ ] Equivalent starts single-flight; all sections begin in bounded parallel and publish ready results atomically/independently. +- [ ] Attempts time out at 60 seconds and only transient failures receive two exponential-backoff/jitter retries. +- [ ] Redis loss yields safe cold recomputation; one-hour events/15-second heartbeat and independent user/merchant limits match specs. + +**Verification:** Queue/cache integration, race, failure, retry timing, Redis-loss, deduplication, replay, rate-limit, and sensitive-payload tests. + +**Dependencies:** F1, F6, and F7. + +**Files likely touched:** `internal/analytics/runs/`, `internal/analytics/jobs/`, Redis adapters, worker root/tests. + +**Estimated scope:** Medium slices + +## Task F9: Implement tenant-scoped detail and export providers + +**Description:** Build the shared authorized query/projection for unverified-paid, failed, and retried details, deterministic keyset pagination, export preflight/count/stream inputs, and operator audit metadata. + +**Acceptance criteria:** + +- [ ] Every query requires session-derived merchant scope and approved period/filter allowlists. +- [ ] Detail/export projections reconcile to metrics and exclude payer tokens, peer rows, credentials, and internal keys. +- [ ] Stable ordering, cursor inputs, 100,000-row/50-MiB/60-second export caps, and audit metadata are available to the API adapter. + +**Verification:** Tenant/tampering/reconciliation/pagination/snapshot/cap/query-plan/leak tests against PostgreSQL. + +**Dependencies:** F2, F6, and F7. + +**Files likely touched:** `internal/analytics/details/`, cursor/projection package, export preflight, audit/tests. + +**Estimated scope:** Medium slices + +## Task F10: Stabilize provider interfaces and backend integration + +**Description:** Wire Identity, period, analytics-run/event, detail/export, audit, PostgreSQL, and Redis providers behind narrow interfaces consumed by the Merchant API. + +**Acceptance criteria:** + +- [ ] Composition roots wire explicit constructors with bounded pools/timeouts and graceful lifecycle. +- [ ] Provider contracts match OpenAPI semantics without importing HTTP types into domain packages. +- [ ] A non-HTTP integration harness completes seed/login/period/run/partial-result/detail/export-provider flows for two isolated merchants. + +**Verification:** Full provider integration, race, shutdown, dependency-loss, tenant-boundary, and safe-log tests. + +**Dependencies:** F4–F9. + +**Files likely touched:** provider interfaces/adapters, `cmd/api/`, `cmd/worker/`, integration tests. + +**Estimated scope:** Medium (split wiring and integration) + +## Checkpoint F: Backend provider readiness + +- [ ] Tasks F1–F10 and every foundation acceptance criterion pass. +- [ ] Full pinned import, two-merchant identity, metric reconciliation, Redis-loss, and tenant isolation evidence is reviewed. +- [ ] Human approves provider readiness before completing HTTP feature slices. ## Task 1: Establish the OpenAPI toolchain and conformance harness @@ -22,7 +208,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D - [ ] Run focused generator/conformance tests and confirm a deliberate contract mismatch fails. - [ ] Review generated and lockfile/checksum diffs; approve any new dependency before installation. -**Dependencies:** Approved implementation authorization and separately approved Go scaffold. +**Dependencies:** Explicit implementation authorization and F1. **Files likely touched:** @@ -50,7 +236,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D - [ ] Cross-tenant client inputs cannot create or replace merchant context. - [ ] `go test ./...` and `go vet ./...` pass. -**Dependencies:** Task 1 and approved Identity session-verification interface. +**Dependencies:** Task 1 and F6 provider interface. **Files likely touched:** @@ -83,7 +269,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D - [ ] OpenAPI conformance tests cover all auth statuses and headers. - [ ] `go test -race ./...` passes for affected packages. -**Dependencies:** Task 2 and approved Identity provider implementation. +**Dependencies:** Task 2 and F6. **Files likely touched:** @@ -110,7 +296,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D - [ ] Integration tests reconcile returned ids/boundaries with `calendar_dates`. - [ ] OpenAPI examples and `go test ./...` pass. -**Dependencies:** Task 2 and approved calendar/period provider implementation. +**Dependencies:** Task 2 and F7. **Files likely touched:** @@ -132,11 +318,11 @@ Every task must satisfy its acceptance criteria and the standing Definition of D **Verification:** -- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, current revisions, and no duplicate jobs. +- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, the immutable dataset snapshot, and no duplicate jobs. - [ ] Contract tests cover `200`, `202`, `401`, `403`, `404`, `422`, `429`, and dependency failures. - [ ] Integration tests assert that rejected/limited requests enqueue no work. -**Dependencies:** Tasks 1–2 and approved Analytics run provider implementation. +**Dependencies:** Tasks 1–2 and F8. **Files likely touched:** @@ -170,7 +356,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D - [ ] Proxy-style tests assert no buffering/compression and required stream headers. - [ ] Race tests cover concurrent publish/replay/disconnect paths. -**Dependencies:** Task 5 and approved Analytics event provider implementation. +**Dependencies:** Task 5 and F8. **Files likely touched:** @@ -187,13 +373,13 @@ Every task must satisfy its acceptance criteria and the standing Definition of D **Acceptance criteria:** -- [ ] Only an exhausted, retryable failed section for the current run/revisions is accepted. +- [ ] Only an exhausted, retryable failed section for the current run/dataset snapshot is accepted. - [ ] Retry idempotency and independent user/merchant throttles prevent duplicate work. - [ ] Ready siblings remain unchanged and stale/cross-tenant requests fail safely. **Verification:** -- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale revisions, throttling, cross-tenant ids, and no sibling restart. +- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale dataset snapshots, throttling, cross-tenant ids, and no sibling restart. - [ ] Rejected retries enqueue no work. - [ ] OpenAPI conformance and `go test -race ./...` pass. @@ -229,7 +415,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D - [ ] Integration queries reconcile representative merchants and approved periods. - [ ] Sensitive-field absence and measured query-plan checks pass. -**Dependencies:** Tasks 2 and 4 plus approved Analytics detail provider implementation. +**Dependencies:** Tasks 2 and 4 plus F9. **Files likely touched:**