211 lines
13 KiB
Markdown
211 lines
13 KiB
Markdown
# Implementation Plan: ZarinPal Merchant Analytics Backend
|
||
|
||
- Status: Approved — Phase 4 implementation authorized 2026-08-20
|
||
- Capabilities: Data foundation, demo identity, analytics engine, and Merchant API
|
||
- Approved contracts: physical schema, import workflow/validation, Identity, Analytics, metric catalogue `0.4.0`, API, and OpenAPI `1.0.0`
|
||
- Task list: `tasks/todo.md`
|
||
- Last updated: 2026-08-20
|
||
|
||
## Overview
|
||
|
||
Build the complete Go capability-oriented modular monolith needed by the dashboard: reviewed PostgreSQL migrations and normalized entities, the guarded one-time importer, demo merchant identity, deterministic analytics queries, Redis/Asynq orchestration, and the approved `/api/v1` REST/SSE/detail/CSV adapter. Deployment configuration remains owned by the deployment repository.
|
||
|
||
## Preconditions and boundaries
|
||
|
||
- The approved Identity, Data Foundation, physical schema, importer, Analytics Engine, API, ADR, metric, and data-guide decisions remain authoritative.
|
||
- Architecture and dependency injection are already approved: capability-oriented modular monolith with manual constructor wiring in thin `cmd/` composition roots.
|
||
- The user approved this plan and task list and authorized implementation on 2026-08-20.
|
||
- Implementation proceeds through `tasks/todo.md` in dependency order, starting with Task F1.
|
||
- The API work depends on implementation-ready provider interfaces from Identity and Analytics. Internal provider signatures may be refined without changing the approved HTTP contract.
|
||
- Any OpenAPI generator, validator, router, rate-limiter, or CSV dependency must be version-pinned, checked against official documentation, security-reviewed, and approved before addition.
|
||
|
||
## Architecture decisions
|
||
|
||
- `docs/openapi.yaml` remains the public interface source of truth. Generated artifacts are reproducible outputs, never a competing hand-written contract.
|
||
- HTTP handlers live in the Merchant API capability and depend on narrow consumer-owned Identity/Analytics interfaces.
|
||
- Shared HTTP middleware owns request ids, request-size limits, safe Problem Details serialization, authentication, CSRF/origin enforcement, and no-store/security headers.
|
||
- Merchant identity is never accepted from a path, query, header, request body, cursor, or event id. It is resolved from the current server-side session on every protected request.
|
||
- JSON detail and CSV export share one canonical filter/authorization/projection policy so export cannot widen the authorized view.
|
||
- SSE reauthorizes connection and replay, uses retained opaque event ids, and resets to a current snapshot when replay is unavailable.
|
||
- Audit and observability record safe metadata and correlation ids, not payment rows, credentials, tokens, raw cursors, or peer data.
|
||
|
||
## Dependency graph
|
||
|
||
```text
|
||
Go scaffold/tooling
|
||
|
|
||
v
|
||
Ent model + reviewed Atlas migrations
|
||
|
|
||
+--> pgx staging importer --> validation/promotion/recovery
|
||
| |
|
||
| v
|
||
| sole immutable dataset snapshot
|
||
|
|
||
+--> demo identity/session service
|
||
|
|
||
+--> calendar + metric query kernel
|
||
|
|
||
v
|
||
Redis/Asynq run orchestration
|
||
|
|
||
v
|
||
Approved OpenAPI --> HTTP boundary --> auth/period/run/SSE/detail/CSV
|
||
|
|
||
v
|
||
audit + full verification
|
||
```
|
||
|
||
## Ordered task index
|
||
|
||
### Foundation phase A: repository and schema
|
||
|
||
- [ ] Task F1: Establish the Go workspace, command roots, configuration, and quality toolchain.
|
||
- [ ] Task F2: Implement the normalized Ent model and reviewed Atlas migration baseline.
|
||
|
||
### Checkpoint F-A: Schema foundation
|
||
|
||
- [ ] Clean migrations create the approved physical schema and ERD relationships without triggers or revision tables.
|
||
- [ ] Build, unit-test, lint, vulnerability, and migration commands are reproducible.
|
||
|
||
### Foundation phase B: one-time dataset bootstrap
|
||
|
||
- [ ] Task F3: Implement pinned archive preflight and bounded pgx staging.
|
||
- [ ] Task F4: Implement set-based validation, atomic promotion, redacted reports, and recovery.
|
||
- [ ] Task F5: Implement deterministic synthetic merchant profiles and secure demo-account seeding.
|
||
|
||
### Checkpoint F-B: Trusted dataset
|
||
|
||
- [ ] The full pinned archive reconciles to approved counts and the second identical run is a no-op.
|
||
- [ ] Fatal input, interruption, concurrency, and sensitive-data leak tests pass.
|
||
|
||
### Foundation phase C: identity and analytics providers
|
||
|
||
- [ ] Task F6: Implement password authentication and revocable server-side merchant sessions.
|
||
- [ ] Task F7: Implement calendar periods and the deterministic metric query kernel.
|
||
- [ ] Task F8: Implement Redis/Asynq analytics runs, section cache, retries, and event history.
|
||
- [ ] Task F9: Implement tenant-scoped detail queries and shared export projection.
|
||
- [ ] Task F10: Stabilize provider interfaces and complete backend-provider integration checks.
|
||
|
||
### Checkpoint F-C: Provider readiness
|
||
|
||
- [ ] Identity, periods, metrics, peer suppression, runs, events, detail, and export providers pass PostgreSQL/Redis integration tests.
|
||
- [ ] Provider contracts expose the immutable dataset snapshot and no client-supplied merchant authority.
|
||
|
||
### API phase 1: Contract and HTTP boundary
|
||
|
||
- [ ] Task 1: Establish the OpenAPI toolchain and conformance harness.
|
||
- [ ] Task 2: Build the shared HTTP boundary and protected-request middleware.
|
||
|
||
### Checkpoint 1: Boundary readiness
|
||
|
||
- [ ] OpenAPI validation/generation is reproducible from a clean checkout.
|
||
- [ ] Shared Problem Details, request limits, request ids, cookie authentication, and CSRF/origin behavior pass focused tests.
|
||
- [ ] Human review confirms no application handler can obtain tenant authority from client input.
|
||
|
||
### API phase 2: Core vertical slices
|
||
|
||
- [ ] Task 3: Implement login, current-session, and logout endpoints.
|
||
- [ ] Task 4: Implement approved reporting-period discovery.
|
||
- [ ] Task 5: Implement analytics-run start and snapshot retrieval.
|
||
|
||
### Checkpoint 2: Core API flow
|
||
|
||
- [ ] A merchant can authenticate, obtain an approved period, start one deduplicated run, and read its snapshot.
|
||
- [ ] Cross-tenant ids are indistinguishable from missing ids.
|
||
- [ ] OpenAPI examples and focused integration tests pass.
|
||
|
||
### API phase 3: Partial delivery and recovery
|
||
|
||
- [ ] Task 6: Implement authenticated SSE delivery and recovery.
|
||
- [ ] Task 7: Implement exhausted-section manual retry.
|
||
|
||
### Checkpoint 3: Asynchronous behavior
|
||
|
||
- [ ] Independent section completion, heartbeat, replay, snapshot reset, revocation, and sibling-failure isolation pass integration tests.
|
||
- [ ] Existing start/retry/SSE limits reject before queue or stream work.
|
||
|
||
### API phase 4: Evidence and export
|
||
|
||
- [ ] Task 8: Implement cursor-paginated session detail views.
|
||
- [ ] Task 9: Implement safeguarded Persian CSV export.
|
||
|
||
### Checkpoint 4: Reconciliation and data boundary
|
||
|
||
- [ ] Each detail view reconciles with its approved aggregate.
|
||
- [ ] JSON and CSV use identical tenant/filter predicates and approved field projections.
|
||
- [ ] Payer tokens, credentials, peer rows, and internal keys are absent from responses, exports, logs, and traces.
|
||
|
||
### API phase 5: Operational hardening
|
||
|
||
- [ ] Task 10: Publish the stable API problem-type catalogue.
|
||
- [ ] Task 11: Add API audit and observability instrumentation.
|
||
- [ ] Task 12: Complete contract, security, race, and performance verification.
|
||
|
||
### Checkpoint 5: Implementation-ready completion
|
||
|
||
- [ ] All OpenAPI operations and documented status codes have passing contract tests.
|
||
- [ ] `go test`, race detection, `go vet`, `golangci-lint`, and `govulncheck` pass using repository-approved commands.
|
||
- [ ] Runtime checks cover tenant isolation, session revocation, SSE recovery, cursor integrity, export caps, formula neutralization, and sensitive-field absence.
|
||
- [ ] Documentation matches runtime behavior and the full project Definition of Done is satisfied.
|
||
- [ ] Human review approves completion before merge or deployment.
|
||
|
||
## Verification strategy
|
||
|
||
### Per task
|
||
|
||
- Write tests that fail before the behavior exists and pass after it is implemented.
|
||
- Run focused package tests for the changed slice.
|
||
- For schema/import slices, compare migrations to the approved ERD, exercise rollback/recovery, and reconcile pinned archive counts without committing source data.
|
||
- For metric slices, compare results to an independent SQL oracle across Tehran/Jalali boundaries, missing values, status semantics, peers, and large/skewed merchants.
|
||
- For identity slices, exercise enumeration resistance, two-session enforcement, expiry/revocation, CSRF, throttling, tenant isolation, and sensitive-value absence.
|
||
- For Redis/Asynq slices, exercise single-flight, independent partial completion, bounded retries, replay, rate limits, and safe cold recomputation after Redis loss.
|
||
- Validate `docs/openapi.yaml` and regenerate/check generated artifacts without diff.
|
||
- Run `go test ./...` and `go vet ./...` once the Go scaffold exists.
|
||
- Inspect the staged diff and scan it for secrets and forbidden data fields before every commit.
|
||
|
||
### At checkpoints
|
||
|
||
- Run `go test -race ./...`, `golangci-lint run`, and `govulncheck ./...` after the required tools are pinned by the approved scaffold/tooling task.
|
||
- Exercise the actual HTTP server against PostgreSQL and Redis test dependencies for authentication, tenant isolation, analytics lifecycle, SSE, pagination, and export behavior.
|
||
- Run OpenAPI request/response conformance tests for all examples, errors, media types, headers, and security requirements.
|
||
- Verify query plans and bounded resource behavior for detail/export paths with representative merchant volumes.
|
||
|
||
## Parallelization
|
||
|
||
- F2 follows F1. After F2, importer preflight/staging (F3), metric-query work (F7), and the non-credential portion of synthetic profile design (F5) may proceed independently.
|
||
- F4 follows F3. Credential seeding completes only after F4, and session work (F6) follows F5.
|
||
- Run orchestration (F8) and detail/export providers (F9) may proceed in parallel after their identity and metric dependencies are stable; F10 integrates them sequentially.
|
||
- API contract tooling (Task 1) may proceed alongside provider work after F1, but protected endpoints wait for their named F-provider dependencies.
|
||
- Tasks 3 and 4 may proceed in parallel only after Tasks 1–2 stabilize shared HTTP types and middleware.
|
||
- Task 6 and the read-only parts of Task 8 may proceed in parallel after Task 5 and the Analytics provider interfaces stabilize.
|
||
- Task 9 is sequential after Task 8 because CSV must reuse its authorization, filters, and projection.
|
||
- Tasks 10–11 may proceed alongside later feature slices once stable error codes and audit events exist.
|
||
- Task 12 is sequential after all feature slices and instrumentation are complete.
|
||
|
||
## Risks and mitigations
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|---|---|---|
|
||
| Import accepts corrupt/oversized data or partially promotes | Critical | Pinned preflight, bounded streaming, set-based validation, atomic promotion, exact reconciliation, fault-injection and recovery tests |
|
||
| Physical schema or metrics drift from approved semantics | Critical | ERD/migration comparison, independent SQL oracle, boundary fixtures, catalogue-version evidence |
|
||
| Demo authentication consumes effort or becomes production identity | Medium | Keep the bounded demo-only session scope and preserve the explicit future ZarinPal SSO replacement boundary |
|
||
| Generated types drift from OpenAPI | High | Reproducible pinned generation plus clean-diff and contract-test gates |
|
||
| Tenant scope leaks through ids/cursors/replay | Critical | Session-derived merchant context, ownership checks in providers, identical not-found policy, adversarial integration tests |
|
||
| SSE behavior differs across proxy/browser boundaries | High | Test event framing, heartbeat, buffering headers, replay, reset, disconnect, and revocation through the deployed-style proxy path |
|
||
| CSV widens detail access or leaks formulas/sensitive data | Critical | Reuse one canonical detail query/projection, preflight limits, formula neutralization, allowlist tests, abort-aware audit result |
|
||
| Retry/idempotency races duplicate jobs | High | Atomic intent claim, canonical request hash, current-run deduplication, concurrency tests |
|
||
| Large merchant queries exhaust API resources | High | Cursor pagination, fixed filters, timeouts, export preflight caps, measured plans, bounded streaming |
|
||
| Error details expose internals | Medium | Central RFC 9457 mapper with stable allowlisted codes and negative leak tests |
|
||
| Pre-1.0 queue behavior changes | High | Preserve approved Asynq `v0.26.0` pin and run retry/deduplication/recovery integration tests before any upgrade |
|
||
|
||
## Implementation gates, not product questions
|
||
|
||
- Select and approve exact pinned OpenAPI generation/validation tooling before adding dependencies.
|
||
- Confirm repository-wide build/test/lint command wrappers when the authorized Go scaffold is created.
|
||
- Confirm the hosting mechanism that makes `https://zarinpal.amiirkhl.ir/docs/errors/{error-slug}` dereferenceable; the URI namespace and content responsibility are already approved.
|
||
|
||
## Product questions
|
||
|
||
None.
|