zarinpal-analytics-backend/tasks/plan.md

Implementation Plan: ZarinPal Merchant Analytics Backend

• Status: Approved — Phase 4 implementation authorized 2026-08-20
• Capabilities: Data foundation, demo identity, analytics engine, and Merchant API
• Approved contracts: physical schema, import workflow/validation, Identity, Analytics, metric catalogue 0.4.0, API, and OpenAPI 1.0.0
• Task list: tasks/todo.md
• Last updated: 2026-08-20

Overview

Build the complete Go capability-oriented modular monolith needed by the dashboard: reviewed PostgreSQL migrations and normalized entities, the guarded one-time importer, demo merchant identity, deterministic analytics queries, Redis/Asynq orchestration, and the approved /api/v1 REST/SSE/detail/CSV adapter. Deployment configuration remains owned by the deployment repository.

Preconditions and boundaries

• The approved Identity, Data Foundation, physical schema, importer, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative.
• Architecture and dependency injection are already approved: capability-oriented modular monolith with manual constructor wiring in thin cmd/ composition roots.
• The user approved this plan and task list and authorized implementation on 2026-08-20.
• Implementation proceeds through tasks/todo.md in dependency order, starting with Task F1.
• The API work depends on implementation-ready provider interfaces from Identity and Analytics. Internal provider signatures may be refined without changing the approved HTTP contract.
• Any OpenAPI generator, validator, router, rate-limiter, or CSV dependency must be version-pinned, checked against official documentation, security-reviewed, and approved before addition.

Architecture decisions

• docs/openapi.yaml remains the public interface source of truth. Generated artifacts are reproducible outputs, never a competing hand-written contract.
• HTTP handlers live in the Merchant API capability and depend on narrow consumer-owned Identity/Analytics interfaces.
• Shared HTTP middleware owns request ids, request-size limits, safe Problem Details serialization, authentication, CSRF/origin enforcement, and no-store/security headers.
• Merchant identity is never accepted from a path, query, header, request body, cursor, or event id. It is resolved from the current server-side session on every protected request.
• JSON detail and CSV export share one canonical filter/authorization/projection policy so export cannot widen the authorized view.
• SSE reauthorizes connection and replay, uses retained opaque event ids, and resets to a current snapshot when replay is unavailable.
• Audit and observability record safe metadata and correlation ids, not payment rows, credentials, tokens, raw cursors, or peer data.

Dependency graph

Go scaffold/tooling
        |
        v
Ent model + reviewed Atlas migrations
        |
        +--> pgx staging importer --> validation/promotion/recovery
        |                                  |
        |                                  v
        |                         sole immutable dataset snapshot
        |
        +--> demo identity/session service
        |
        +--> calendar + metric query kernel
                         |
                         v
                 Redis/Asynq run orchestration
                         |
                         v
Approved OpenAPI --> HTTP boundary --> auth/period/run/SSE/detail/CSV
                                         |
                                         v
                              audit + full verification

Ordered task index

Foundation phase A: repository and schema

• Task F1: Establish the Go workspace, command roots, configuration, and quality toolchain.
• Task F2: Implement the normalized Ent model and reviewed Atlas migration baseline.

Checkpoint F-A: Schema foundation

• Clean migrations create the approved physical schema and ERD relationships without triggers or revision tables.
• Build, unit-test, lint, vulnerability, and migration commands are reproducible.

Foundation phase B: one-time dataset bootstrap

• Task F3: Implement pinned archive preflight and bounded pgx staging.
• Task F4: Implement set-based validation, atomic promotion, redacted reports, and recovery.
• Task F5: Implement deterministic synthetic merchant profiles and secure demo-account seeding.

Checkpoint F-B: Trusted dataset

• The full pinned archive reconciles to approved counts and the second identical run is a no-op.
• Fatal input, interruption, concurrency, and sensitive-data leak tests pass.

Foundation phase C: identity and analytics providers

• Task F6: Implement password authentication and revocable server-side merchant sessions.
• Task F7: Implement calendar periods and the deterministic metric query kernel.
• Task F8: Implement Redis/Asynq analytics runs, section cache, retries, and event history.
• Task F9: Implement tenant-scoped detail queries and shared export projection.
• Task F10: Stabilize provider interfaces and complete backend-provider integration checks.

Checkpoint F-C: Provider readiness

• Identity, periods, metrics, peer suppression, runs, events, detail, and export providers pass PostgreSQL/Redis integration tests.
• Provider contracts expose the immutable dataset snapshot and no client-supplied merchant authority.

API phase 1: Contract and HTTP boundary

• Task 1: Establish the OpenAPI toolchain and conformance harness.
• Task 2: Build the shared HTTP boundary and protected-request middleware.

Checkpoint 1: Boundary readiness

• OpenAPI validation/generation is reproducible from a clean checkout.
• Shared Problem Details, request limits, request ids, cookie authentication, and CSRF/origin behavior pass focused tests.
• Human review confirms no application handler can obtain tenant authority from client input.

API phase 2: Core vertical slices

• Task 3: Implement login, current-session, and logout endpoints.
• Task 4: Implement approved reporting-period discovery.
• Task 5: Implement analytics-run start and snapshot retrieval.

Checkpoint 2: Core API flow

• A merchant can authenticate, obtain an approved period, start one deduplicated run, and read its snapshot.
• Cross-tenant ids are indistinguishable from missing ids.
• OpenAPI examples and focused integration tests pass.

API phase 3: Partial delivery and recovery

• Task 6: Implement authenticated SSE delivery and recovery.
• Task 7: Implement exhausted-section manual retry.

Checkpoint 3: Asynchronous behavior

• Independent section completion, heartbeat, replay, snapshot reset, revocation, and sibling-failure isolation pass integration tests.
• Existing start/retry/SSE limits reject before queue or stream work.

API phase 4: Evidence and export

• Task 8: Implement cursor-paginated session detail views.
• Task 9: Implement safeguarded Persian CSV export.

Checkpoint 4: Reconciliation and data boundary

• Each detail view reconciles with its approved aggregate.
• JSON and CSV use identical tenant/filter predicates and approved field projections.
• Payer tokens, credentials, peer rows, and internal keys are absent from responses, exports, logs, and traces.

API phase 5: Operational hardening

• Task 10: Publish the stable API problem-type catalogue.
• Task 11: Add API audit and observability instrumentation.
• Task 12: Complete contract, security, race, and performance verification.

Checkpoint 5: Implementation-ready completion

• All OpenAPI operations and documented status codes have passing contract tests.
• go test, race detection, go vet, golangci-lint, and govulncheck pass using repository-approved commands.
• Runtime checks cover tenant isolation, session revocation, SSE recovery, cursor integrity, export caps, formula neutralization, and sensitive-field absence.
• Documentation matches runtime behavior and the full project Definition of Done is satisfied.
• Human review approves completion before merge or deployment.

Verification strategy

Per task

• Write tests that fail before the behavior exists and pass after it is implemented.
• Run focused package tests for the changed slice.
• For schema/import slices, compare migrations to the approved ERD, exercise rollback/recovery, and reconcile pinned archive counts without committing source data.
• For metric slices, compare results to an independent SQL oracle across Tehran/Jalali boundaries, missing values, status semantics, peers, and large/skewed merchants.
• For identity slices, exercise enumeration resistance, two-session enforcement, expiry/revocation, CSRF, throttling, tenant isolation, and sensitive-value absence.
• For Redis/Asynq slices, exercise single-flight, independent partial completion, bounded retries, replay, rate limits, and safe cold recomputation after Redis loss.
• Validate docs/openapi.yaml and regenerate/check generated artifacts without diff.
• Run go test ./... and go vet ./... once the Go scaffold exists.
• Inspect the staged diff and scan it for secrets and forbidden data fields before every commit.

At checkpoints

• Run go test -race ./..., golangci-lint run, and govulncheck ./... after the required tools are pinned by the approved scaffold/tooling task.
• Exercise the actual HTTP server against PostgreSQL and Redis test dependencies for authentication, tenant isolation, analytics lifecycle, SSE, pagination, and export behavior.
• Run OpenAPI request/response conformance tests for all examples, errors, media types, headers, and security requirements.
• Verify query plans and bounded resource behavior for detail/export paths with representative merchant volumes.

Parallelization

• F2 follows F1. After F2, importer preflight/staging (F3), metric-query work (F7), and the non-credential portion of synthetic profile design (F5) may proceed independently.
• F4 follows F3. Credential seeding completes only after F4, and session work (F6) follows F5.
• Run orchestration (F8) and detail/export providers (F9) may proceed in parallel after their identity and metric dependencies are stable; F10 integrates them sequentially.
• API contract tooling (Task 1) may proceed alongside provider work after F1, but protected endpoints wait for their named F-provider dependencies.
• Tasks 3 and 4 may proceed in parallel only after Tasks 1–2 stabilize shared HTTP types and middleware.
• Task 6 and the read-only parts of Task 8 may proceed in parallel after Task 5 and the Analytics provider interfaces stabilize.
• Task 9 is sequential after Task 8 because CSV must reuse its authorization, filters, and projection.
• Tasks 10–11 may proceed alongside later feature slices once stable error codes and audit events exist.
• Task 12 is sequential after all feature slices and instrumentation are complete.

Risks and mitigations

Risk	Impact	Mitigation
Import accepts corrupt/oversized data or partially promotes	Critical	Pinned preflight, bounded streaming, set-based validation, atomic promotion, exact reconciliation, fault-injection and recovery tests
Physical schema or metrics drift from approved semantics	Critical	ERD/migration comparison, independent SQL oracle, boundary fixtures, catalogue-version evidence
Demo authentication consumes effort or becomes production identity	Medium	Keep the bounded demo-only session scope and preserve the explicit future ZarinPal SSO replacement boundary
Generated types drift from OpenAPI	High	Reproducible pinned generation plus clean-diff and contract-test gates
Tenant scope leaks through ids/cursors/replay	Critical	Session-derived merchant context, ownership checks in providers, identical not-found policy, adversarial integration tests
SSE behavior differs across proxy/browser boundaries	High	Test event framing, heartbeat, buffering headers, replay, reset, disconnect, and revocation through the deployed-style proxy path
CSV widens detail access or leaks formulas/sensitive data	Critical	Reuse one canonical detail query/projection, preflight limits, formula neutralization, allowlist tests, abort-aware audit result
Retry/idempotency races duplicate jobs	High	Atomic intent claim, canonical request hash, current-run deduplication, concurrency tests
Large merchant queries exhaust API resources	High	Cursor pagination, fixed filters, timeouts, export preflight caps, measured plans, bounded streaming
Error details expose internals	Medium	Central RFC 9457 mapper with stable allowlisted codes and negative leak tests
Pre-1.0 queue behavior changes	High	Preserve approved Asynq v0.26.0 pin and run retry/deduplication/recovery integration tests before any upgrade

Implementation gates, not product questions

• Select and approve exact pinned OpenAPI generation/validation tooling before adding dependencies.
• Confirm repository-wide build/test/lint command wrappers when the authorized Go scaffold is created.
• Confirm the hosting mechanism that makes https://zarinpal.amiirkhl.ir/docs/errors/{error-slug} dereferenceable; the URI namespace and content responsibility are already approved.

Product questions

None.