starcoach-hackathon/zarinpal-analytics-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plan): expand backend implementation roadmap

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-08-20 19:00:07 +03:30
parent ec625252fb
commit b82ba0dc23
2 changed files with 277 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

116
tasks/plan.md
View File

@@ -1,18 +1,18 @@
# Implementation Plan: Merchant Analytics API
# Implementation Plan: ZarinPal Merchant Analytics Backend
- Status: Approved plan and task breakdown — implementation not authorized
- Capability: Merchant API
- Approved contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0`
- Status: Expanded implementation plan ready for stakeholder review — implementation not authorized
- Capabilities: Data foundation, demo identity, analytics engine, and Merchant API
- Approved contracts: physical schema, import workflow/validation, Identity, Analytics, metric catalogue `0.4.0`, API, and OpenAPI `1.0.0`
- Task list: `tasks/todo.md`
- Last updated: 2026-08-20
## Overview
Implement the approved `/api/v1` REST, SSE, paginated-detail, and CSV interfaces as the HTTP adapter of the capability-oriented Go modular monolith. The API layer validates untrusted HTTP input, resolves the merchant from the server-side session, calls approved Identity/Analytics provider contracts, and serializes only OpenAPI-defined output. It does not own metric formulas, normalized payment facts, queue behavior, or deployment configuration.
Build the complete Go capability-oriented modular monolith needed by the dashboard: reviewed PostgreSQL migrations and normalized entities, the guarded one-time importer, demo merchant identity, deterministic analytics queries, Redis/Asynq orchestration, and the approved `/api/v1` REST/SSE/detail/CSV adapter. Deployment configuration remains owned by the deployment repository.
## Preconditions and boundaries
- The approved Identity, Data Foundation, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative.
- The approved Identity, Data Foundation, physical schema, importer, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative.
- Architecture and dependency injection are already approved: capability-oriented modular monolith with manual constructor wiring in thin `cmd/` composition roots.
- This plan does not authorize `go.mod`, packages, generated files, dependencies, migrations, containers, or executable code.
- Implementation starts only after the plan and task list are approved and the user separately authorizes implementation.
@@ -32,38 +32,67 @@ Implement the approved `/api/v1` REST, SSE, paginated-detail, and CSV interfaces
## Dependency graph
```text
Approved OpenAPI 1.0.0 + backend scaffold
|
v
Contract toolchain and HTTP boundary
|
+---------+----------+
| |
v v
Identity adapter Analytics provider adapters
| |
v +---------+---------+
Auth endpoints Periods Runs/snapshot
|
+------+------+
v v
SSE Section retry
Analytics provider adapters
|
v
Detail JSON pagination
|
v
CSV export
|
v
Audit, observability, and full conformance
Go scaffold/tooling
|
v
Ent model + reviewed Atlas migrations
|
+--> pgx staging importer --> validation/promotion/recovery
| |
| v
| sole immutable dataset snapshot
|
+--> demo identity/session service
|
+--> calendar + metric query kernel
|
v
Redis/Asynq run orchestration
|
v
Approved OpenAPI --> HTTP boundary --> auth/period/run/SSE/detail/CSV
|
v
audit + full verification
```
## Ordered task index
### Phase 1: Contract and HTTP boundary
### Foundation phase A: repository and schema
- [ ] Task F1: Establish the Go workspace, command roots, configuration, and quality toolchain.
- [ ] Task F2: Implement the normalized Ent model and reviewed Atlas migration baseline.
### Checkpoint F-A: Schema foundation
- [ ] Clean migrations create the approved physical schema and ERD relationships without triggers or revision tables.
- [ ] Build, unit-test, lint, vulnerability, and migration commands are reproducible.
### Foundation phase B: one-time dataset bootstrap
- [ ] Task F3: Implement pinned archive preflight and bounded pgx staging.
- [ ] Task F4: Implement set-based validation, atomic promotion, redacted reports, and recovery.
- [ ] Task F5: Implement deterministic synthetic merchant profiles and secure demo-account seeding.
### Checkpoint F-B: Trusted dataset
- [ ] The full pinned archive reconciles to approved counts and the second identical run is a no-op.
- [ ] Fatal input, interruption, concurrency, and sensitive-data leak tests pass.
### Foundation phase C: identity and analytics providers
- [ ] Task F6: Implement password authentication and revocable server-side merchant sessions.
- [ ] Task F7: Implement calendar periods and the deterministic metric query kernel.
- [ ] Task F8: Implement Redis/Asynq analytics runs, section cache, retries, and event history.
- [ ] Task F9: Implement tenant-scoped detail queries and shared export projection.
- [ ] Task F10: Stabilize provider interfaces and complete backend-provider integration checks.
### Checkpoint F-C: Provider readiness
- [ ] Identity, periods, metrics, peer suppression, runs, events, detail, and export providers pass PostgreSQL/Redis integration tests.
- [ ] Provider contracts expose the immutable dataset snapshot and no client-supplied merchant authority.
### API phase 1: Contract and HTTP boundary
- [ ] Task 1: Establish the OpenAPI toolchain and conformance harness.
- [ ] Task 2: Build the shared HTTP boundary and protected-request middleware.
@@ -74,7 +103,7 @@ Analytics provider adapters
- [ ] Shared Problem Details, request limits, request ids, cookie authentication, and CSRF/origin behavior pass focused tests.
- [ ] Human review confirms no application handler can obtain tenant authority from client input.
### Phase 2: Core vertical slices
### API phase 2: Core vertical slices
- [ ] Task 3: Implement login, current-session, and logout endpoints.
- [ ] Task 4: Implement approved reporting-period discovery.
@@ -86,7 +115,7 @@ Analytics provider adapters
- [ ] Cross-tenant ids are indistinguishable from missing ids.
- [ ] OpenAPI examples and focused integration tests pass.
### Phase 3: Partial delivery and recovery
### API phase 3: Partial delivery and recovery
- [ ] Task 6: Implement authenticated SSE delivery and recovery.
- [ ] Task 7: Implement exhausted-section manual retry.
@@ -96,7 +125,7 @@ Analytics provider adapters
- [ ] Independent section completion, heartbeat, replay, snapshot reset, revocation, and sibling-failure isolation pass integration tests.
- [ ] Existing start/retry/SSE limits reject before queue or stream work.
### Phase 4: Evidence and export
### API phase 4: Evidence and export
- [ ] Task 8: Implement cursor-paginated session detail views.
- [ ] Task 9: Implement safeguarded Persian CSV export.
@@ -107,7 +136,7 @@ Analytics provider adapters
- [ ] JSON and CSV use identical tenant/filter predicates and approved field projections.
- [ ] Payer tokens, credentials, peer rows, and internal keys are absent from responses, exports, logs, and traces.
### Phase 5: Operational hardening
### API phase 5: Operational hardening
- [ ] Task 10: Publish the stable API problem-type catalogue.
- [ ] Task 11: Add API audit and observability instrumentation.
@@ -127,6 +156,10 @@ Analytics provider adapters
- Write tests that fail before the behavior exists and pass after it is implemented.
- Run focused package tests for the changed slice.
- For schema/import slices, compare migrations to the approved ERD, exercise rollback/recovery, and reconcile pinned archive counts without committing source data.
- For metric slices, compare results to an independent SQL oracle across Tehran/Jalali boundaries, missing values, status semantics, peers, and large/skewed merchants.
- For identity slices, exercise enumeration resistance, two-session enforcement, expiry/revocation, CSRF, throttling, tenant isolation, and sensitive-value absence.
- For Redis/Asynq slices, exercise single-flight, independent partial completion, bounded retries, replay, rate limits, and safe cold recomputation after Redis loss.
- Validate `docs/openapi.yaml` and regenerate/check generated artifacts without diff.
- Run `go test ./...` and `go vet ./...` once the Go scaffold exists.
- Inspect the staged diff and scan it for secrets and forbidden data fields before every commit.
@@ -140,6 +173,10 @@ Analytics provider adapters
## Parallelization
- F2 follows F1. After F2, importer preflight/staging (F3), metric-query work (F7), and the non-credential portion of synthetic profile design (F5) may proceed independently.
- F4 follows F3. Credential seeding completes only after F4, and session work (F6) follows F5.
- Run orchestration (F8) and detail/export providers (F9) may proceed in parallel after their identity and metric dependencies are stable; F10 integrates them sequentially.
- API contract tooling (Task 1) may proceed alongside provider work after F1, but protected endpoints wait for their named F-provider dependencies.
- Tasks 3 and 4 may proceed in parallel only after Tasks 1–2 stabilize shared HTTP types and middleware.
- Task 6 and the read-only parts of Task 8 may proceed in parallel after Task 5 and the Analytics provider interfaces stabilize.
- Task 9 is sequential after Task 8 because CSV must reuse its authorization, filters, and projection.
@@ -150,6 +187,9 @@ Analytics provider adapters
| Risk | Impact | Mitigation |
|---|---|---|
| Import accepts corrupt/oversized data or partially promotes | Critical | Pinned preflight, bounded streaming, set-based validation, atomic promotion, exact reconciliation, fault-injection and recovery tests |
| Physical schema or metrics drift from approved semantics | Critical | ERD/migration comparison, independent SQL oracle, boundary fixtures, catalogue-version evidence |
| Demo authentication consumes effort or becomes production identity | Medium | Keep the bounded demo-only session scope and preserve the explicit future ZarinPal SSO replacement boundary |
| Generated types drift from OpenAPI | High | Reproducible pinned generation plus clean-diff and contract-test gates |
| Tenant scope leaks through ids/cursors/replay | Critical | Session-derived merchant context, ownership checks in providers, identical not-found policy, adversarial integration tests |
| SSE behavior differs across proxy/browser boundaries | High | Test event framing, heartbeat, buffering headers, replay, reset, disconnect, and revocation through the deployed-style proxy path |

212
tasks/todo.md
View File

@@ -1,10 +1,196 @@
# Merchant Analytics API Task List
# ZarinPal Merchant Analytics Backend Task List
- Status: Approved task breakdown — implementation not authorized
- Status: Expanded task breakdown ready for stakeholder review — implementation not authorized
- Plan: `tasks/plan.md`
- Contract: `docs/API-SPEC.md` and `docs/openapi.yaml` version `1.0.0`
Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files.
Every task must satisfy its acceptance criteria and the standing Definition of Done. File lists are expected blast radii, not permission to modify unrelated files. Foundation tasks use `F` identifiers; the previously approved API tasks retain their stable numeric identifiers.
## Task F1: Establish the Go workspace and quality toolchain
**Description:** Create the thin command-root/modular-monolith skeleton, validated configuration boundary, structured logging baseline, and pinned repository quality commands without implementing product behavior.
**Acceptance criteria:**
- [ ] `cmd/api`, `cmd/worker`, `cmd/importer`, and migration/admin composition roots are thin and business packages live under `internal/`.
- [ ] Manual constructor wiring, context propagation, validated environment configuration, and safe structured logs follow approved ADRs/AGENTS.
- [ ] Pinned build/test/race/vet/lint/vulnerability/generation commands are documented and reproducible.
**Verification:** Run every scaffold quality command from a clean checkout; confirm no service starts implicitly and dependency/checksum diffs are approved.
**Dependencies:** Explicit implementation authorization.
**Files likely touched:** `go.mod`, `go.sum`, `cmd/`, `internal/platform/`, quality-tool configuration.
**Estimated scope:** Medium (split command roots if more than five files)
## Task F2: Implement the normalized Ent model and migration baseline
**Description:** Translate the approved physical schema into Ent entities/relationships and reviewed Atlas SQL migrations, including identity, import, payment, calendar, and audit tables without cache-revision triggers.
**Acceptance criteria:**
- [ ] Tables, FKs, checks, unique/partial indexes, nullability, restrictive deletion, and 30/90-day audit retention match the physical schema.
- [ ] `payment_sessions.import_run_id` is required; `NoAttempt` never becomes an attempt; terminal verification method and session-grain money are stored once.
- [ ] Clean creation and supported upgrade migrations are versioned, reviewed, and never run from API startup.
**Verification:** Run zero-to-head/upgrade migration tests, schema introspection/ERD comparison, constraint negatives, and representative query-plan checks.
**Dependencies:** F1.
**Files likely touched:** `internal/data/schema/`, `ent/schema/`, `migrations/`, migration tests, Atlas configuration.
**Estimated scope:** Medium slices by schema group
## Task F3: Implement pinned archive preflight and bounded staging
**Description:** Build the one-shot importer boundary that validates the fixed file/checksum/schema/resource caps and streams strict CSV/gzip records through pgx `COPY` into run-scoped staging.
**Acceptance criteria:**
- [ ] Fixed filename, SHA-256, 22-column schema, UTF-8/gzip integrity, byte/row/record caps, and regular read-only path checks fail closed.
- [ ] Parsing is bounded-memory, independent of file/session order, and records safe row numbers without logging source values.
- [ ] One PostgreSQL advisory lock prevents concurrent validate/import/recovery commands.
**Verification:** Unit/fuzz tests cover malformed CSV/UTF-8/gzip, overflow, traversal/symlink, expansion, cancellation, caps, and redaction; staging load stays within measured memory.
**Dependencies:** F1–F2.
**Files likely touched:** `cmd/importer/`, `internal/importing/preflight/`, `internal/importing/staging/`, focused tests.
**Estimated scope:** Medium, split preflight and staging commits
## Task F4: Implement validation, atomic promotion, reports, and recovery
**Description:** Add set-based relationship validation, warning/fatal classification, exact reconciliation, transactional normalization, redacted reports, staging cleanup, no-op detection, and restart-from-source recovery.
**Acceptance criteria:**
- [ ] Any fatal issue promotes zero normalized facts; documented timestamp/sequence/null anomalies remain unchanged warnings.
- [ ] Pinned counts and per-merchant/status reconciliations pass before the sole import run becomes completed.
- [ ] Success/failure removes raw staging; interrupted recovery cleans only its run scope and restarts from the immutable archive.
**Verification:** Full-archive rehearsal plus relationship/rollback/fault-injection/concurrency/no-op/recovery/leak tests; JSON and Markdown reports reconcile.
**Dependencies:** F3.
**Files likely touched:** `internal/importing/validation/`, `internal/importing/promotion/`, `internal/importing/report/`, recovery command/tests.
**Estimated scope:** Medium slices; never one XL commit
## Task F5: Implement synthetic merchant profiles and demo-account seeding
**Description:** Generate deterministic non-sensitive merchant display profiles and exactly one initial demo user per merchant through a separate secure administrative flow.
**Acceptance criteria:**
- [ ] Display names are deterministic, clearly synthetic in storage, and stable across identical imports.
- [ ] Usernames are unique and credentials use approved Argon2id hashing; plaintext is emitted once only to an explicit protected output.
- [ ] Re-running the seed is idempotent and never derives passwords from merchant/source keys.
**Verification:** Determinism/idempotency/hash/permission/redaction tests and two-merchant credential smoke fixture.
**Dependencies:** F2 and completed F4 provider behavior.
**Files likely touched:** `cmd/admin/`, `internal/identity/seeding/`, profile generator/tests, operator output adapter.
**Estimated scope:** Medium (3–5 files)
## Task F6: Implement password authentication and server-side sessions
**Description:** Implement demo credential verification, generic throttled outcomes, two-active-session enforcement, opaque token digests, CSRF binding, expiry/activity coalescing, logout/revocation, and authentication audit retention.
**Acceptance criteria:**
- [ ] Authentication never reveals account existence and returns tenant context only from a valid server-side session.
- [ ] Cookie/token/CSRF, idle/absolute expiry, revocation, disabled account, and two-session behavior match Identity spec.
- [ ] Raw credentials/tokens/IP/user-agent values are absent from storage, logs, and audit evidence.
**Verification:** Table/race/integration tests cover enumeration, throttles, fixation, rotation, expiry, revocation, concurrency, and 30-day cleanup.
**Dependencies:** F2 and F5.
**Files likely touched:** `internal/identity/`, session repository, throttling/audit adapters, focused tests.
**Estimated scope:** Medium slices
## Task F7: Implement reporting periods and deterministic metric queries
**Description:** Build calendar-period discovery and SQL/Ent query services for catalogue `0.4.0`, preserving session grain, missingness, Saturday–Friday/Jalali/season boundaries, and peer suppression/equal weighting.
**Acceptance criteria:**
- [ ] Every catalogue metric reconciles for day/week/month/season and emits unit, denominator/sample/coverage, dataset snapshot, metric version, and calculation time.
- [ ] `Paid` plus `Verified`, verification completion, retry recovery, timing percentiles, fees, and `NoAttempt` semantics match approved formulas.
- [ ] Peer outputs exclude the active merchant, weight merchants equally, invert lower-is-better percentiles, and suppress below both thresholds.
**Verification:** Independent SQL oracle tests, boundary fixtures, large/skewed merchant fixtures, null/zero/unknown-status tests, and measured query plans.
**Dependencies:** F2 and trusted F4 dataset fixtures.
**Files likely touched:** `internal/analytics/periods/`, `internal/analytics/metrics/`, peer queries, fixtures/tests.
**Estimated scope:** Medium slices by section
## Task F8: Implement Redis/Asynq analytics run orchestration
**Description:** Implement idempotent four-section runs, dataset-versioned 30-day cache keys, Asynq `v0.26.0` jobs, bounded concurrency, retries, rate limits, event history, and sibling-failure isolation.
**Acceptance criteria:**
- [ ] Equivalent starts single-flight; all sections begin in bounded parallel and publish ready results atomically/independently.
- [ ] Attempts time out at 60 seconds and only transient failures receive two exponential-backoff/jitter retries.
- [ ] Redis loss yields safe cold recomputation; one-hour events/15-second heartbeat and independent user/merchant limits match specs.
**Verification:** Queue/cache integration, race, failure, retry timing, Redis-loss, deduplication, replay, rate-limit, and sensitive-payload tests.
**Dependencies:** F1, F6, and F7.
**Files likely touched:** `internal/analytics/runs/`, `internal/analytics/jobs/`, Redis adapters, worker root/tests.
**Estimated scope:** Medium slices
## Task F9: Implement tenant-scoped detail and export providers
**Description:** Build the shared authorized query/projection for unverified-paid, failed, and retried details, deterministic keyset pagination, export preflight/count/stream inputs, and operator audit metadata.
**Acceptance criteria:**
- [ ] Every query requires session-derived merchant scope and approved period/filter allowlists.
- [ ] Detail/export projections reconcile to metrics and exclude payer tokens, peer rows, credentials, and internal keys.
- [ ] Stable ordering, cursor inputs, 100,000-row/50-MiB/60-second export caps, and audit metadata are available to the API adapter.
**Verification:** Tenant/tampering/reconciliation/pagination/snapshot/cap/query-plan/leak tests against PostgreSQL.
**Dependencies:** F2, F6, and F7.
**Files likely touched:** `internal/analytics/details/`, cursor/projection package, export preflight, audit/tests.
**Estimated scope:** Medium slices
## Task F10: Stabilize provider interfaces and backend integration
**Description:** Wire Identity, period, analytics-run/event, detail/export, audit, PostgreSQL, and Redis providers behind narrow interfaces consumed by the Merchant API.
**Acceptance criteria:**
- [ ] Composition roots wire explicit constructors with bounded pools/timeouts and graceful lifecycle.
- [ ] Provider contracts match OpenAPI semantics without importing HTTP types into domain packages.
- [ ] A non-HTTP integration harness completes seed/login/period/run/partial-result/detail/export-provider flows for two isolated merchants.
**Verification:** Full provider integration, race, shutdown, dependency-loss, tenant-boundary, and safe-log tests.
**Dependencies:** F4–F9.
**Files likely touched:** provider interfaces/adapters, `cmd/api/`, `cmd/worker/`, integration tests.
**Estimated scope:** Medium (split wiring and integration)
## Checkpoint F: Backend provider readiness
- [ ] Tasks F1–F10 and every foundation acceptance criterion pass.
- [ ] Full pinned import, two-merchant identity, metric reconciliation, Redis-loss, and tenant isolation evidence is reviewed.
- [ ] Human approves provider readiness before completing HTTP feature slices.
## Task 1: Establish the OpenAPI toolchain and conformance harness
@@ -22,7 +208,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
- [ ] Run focused generator/conformance tests and confirm a deliberate contract mismatch fails.
- [ ] Review generated and lockfile/checksum diffs; approve any new dependency before installation.
**Dependencies:** Approved implementation authorization and separately approved Go scaffold.
**Dependencies:** Explicit implementation authorization and F1.
**Files likely touched:**
@@ -50,7 +236,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
- [ ] Cross-tenant client inputs cannot create or replace merchant context.
- [ ] `go test ./...` and `go vet ./...` pass.
**Dependencies:** Task 1 and approved Identity session-verification interface.
**Dependencies:** Task 1 and F6 provider interface.
**Files likely touched:**
@@ -83,7 +269,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
- [ ] OpenAPI conformance tests cover all auth statuses and headers.
- [ ] `go test -race ./...` passes for affected packages.
**Dependencies:** Task 2 and approved Identity provider implementation.
**Dependencies:** Task 2 and F6.
**Files likely touched:**
@@ -110,7 +296,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
- [ ] Integration tests reconcile returned ids/boundaries with `calendar_dates`.
- [ ] OpenAPI examples and `go test ./...` pass.
**Dependencies:** Task 2 and approved calendar/period provider implementation.
**Dependencies:** Task 2 and F7.
**Files likely touched:**
@@ -132,11 +318,11 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
**Verification:**
- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, current revisions, and no duplicate jobs.
- [ ] Concurrency tests cover atomic idempotency claim, in-flight duplicates, the immutable dataset snapshot, and no duplicate jobs.
- [ ] Contract tests cover `200`, `202`, `401`, `403`, `404`, `422`, `429`, and dependency failures.
- [ ] Integration tests assert that rejected/limited requests enqueue no work.
**Dependencies:** Tasks 1–2 and approved Analytics run provider implementation.
**Dependencies:** Tasks 1–2 and F8.
**Files likely touched:**
@@ -170,7 +356,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
- [ ] Proxy-style tests assert no buffering/compression and required stream headers.
- [ ] Race tests cover concurrent publish/replay/disconnect paths.
**Dependencies:** Task 5 and approved Analytics event provider implementation.
**Dependencies:** Task 5 and F8.
**Files likely touched:**
@@ -187,13 +373,13 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
**Acceptance criteria:**
- [ ] Only an exhausted, retryable failed section for the current run/revisions is accepted.
- [ ] Only an exhausted, retryable failed section for the current run/dataset snapshot is accepted.
- [ ] Retry idempotency and independent user/merchant throttles prevent duplicate work.
- [ ] Ready siblings remain unchanged and stale/cross-tenant requests fail safely.
**Verification:**
- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale revisions, throttling, cross-tenant ids, and no sibling restart.
- [ ] Tests cover retryable/non-retryable states, duplicate keys, stale dataset snapshots, throttling, cross-tenant ids, and no sibling restart.
- [ ] Rejected retries enqueue no work.
- [ ] OpenAPI conformance and `go test -race ./...` pass.
@@ -229,7 +415,7 @@ Every task must satisfy its acceptance criteria and the standing Definition of D
- [ ] Integration queries reconcile representative merchants and approved periods.
- [ ] Sensitive-field absence and measured query-plan checks pass.
**Dependencies:** Tasks 2 and 4 plus approved Analytics detail provider implementation.
**Dependencies:** Tasks 2 and 4 plus F9.
**Files likely touched:**