From 491c67315301dba351965376062ace42fd2f8f20 Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Thu, 20 Aug 2026 13:12:51 +0330 Subject: [PATCH] docs: approve Ent and pgx data access --- AGENTS.md | 2 ++ docs/DATA-FOUNDATION-SPEC.md | 11 ++++---- docs/decisions/0003-ent-pgx-data-access.md | 30 ++++++++++++++++++++++ 3 files changed, 38 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/0003-ent-pgx-data-access.md diff --git a/AGENTS.md b/AGENTS.md index e2a6531..b9ffa64 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,8 @@ For every Go-related task, always invoke and follow the `golang-how-to` skill, t - Configuration comes from validated environment variables. Logs go to stdout as structured records without credentials, card keys, or sensitive payloads. - Propagate `context.Context`; bound external calls, queues, batches, and database pools. - Use parameterized queries and transactions with explicit boundaries. Stream/bulk-load the multi-million-row input instead of holding it in memory. +- Use Ent for application persistence and relationship modeling. Use pgx with PostgreSQL `COPY` for the bulk-import staging path; do not force high-volume ingestion through row-by-row ORM operations. +- Database migrations are explicit, versioned, reviewed artifacts applied by a dedicated migration/deployment step. Never run Ent automatic schema creation against production during application startup. - Passwords use an approved adaptive hash (Argon2id or bcrypt per the approved auth spec); sessions use secure, HttpOnly cookies and server-side revocation. - Required verification includes unit/integration tests, race detection where applicable, `go vet`, `golangci-lint`, `govulncheck`, and migration/import tests. diff --git a/docs/DATA-FOUNDATION-SPEC.md b/docs/DATA-FOUNDATION-SPEC.md index 8695ed4..636ae31 100644 --- a/docs/DATA-FOUNDATION-SPEC.md +++ b/docs/DATA-FOUNDATION-SPEC.md @@ -150,10 +150,11 @@ Resolved: use a capability-oriented modular monolith, with lightweight ports/ada Resolved: use manual constructor injection with explicit wiring at thin `cmd/` composition roots. See ADR 0002. +Resolved: use Ent for application persistence/modeling and pgx/PostgreSQL `COPY` for streamed bulk-ingestion staging. Migrations are explicit and versioned; production startup never performs automatic schema changes. See ADR 0003. + Remaining approval gates: -1. PostgreSQL access stack: ORM choice plus a bulk-ingest path suitable for 2.2M+ rows. -2. Source timestamp timezone and daylight-saving assumptions. -3. Whether `payer_card_key` must be retained, irreversibly transformed, or omitted after derived analytics. -4. Minimum cohort size and suppression policy for peer benchmarks. -5. Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone. +1. Source timestamp timezone and daylight-saving assumptions. +2. Whether `payer_card_key` must be retained, irreversibly transformed, or omitted after derived analytics. +3. Minimum cohort size and suppression policy for peer benchmarks. +4. Physical partitioning/materialization strategy, which must follow measured query needs rather than the logical ERD alone. diff --git a/docs/decisions/0003-ent-pgx-data-access.md b/docs/decisions/0003-ent-pgx-data-access.md new file mode 100644 index 0000000..58fe57e --- /dev/null +++ b/docs/decisions/0003-ent-pgx-data-access.md @@ -0,0 +1,30 @@ +# ADR 0003: Ent application persistence with pgx bulk ingestion + +- Status: Accepted +- Date: 2026-08-20 + +## Context + +The backend requires an ORM-backed relational model with explicit entity relationships and a repeatable path from schema definitions to an ERD. The source archive contains more than 2.2 million payment-attempt rows, making row-by-row ORM insertion unsuitable for ingestion. Production schema changes must remain reviewable and reversible rather than occurring implicitly at process startup. + +## Decision + +Use Ent for application persistence, relationship modeling, and type-safe access within capability adapters. Keep Ent-generated artifacts and the schema definitions that produce them in Git according to the future scaffold specification. + +Use pgx directly for the importer’s PostgreSQL staging boundary. Feed `CopyFrom` with a streaming `CopyFromSource` (or equivalent bounded batches) so decompressed CSV rows are not accumulated in memory. Validate and promote staged data through explicit transactions/steps described by the import spec; bulk loading must not bypass reconciliation or data-quality rules. + +Use versioned SQL migrations generated/planned from the reviewed schema and committed to Git. Apply them through a dedicated migration command or deployment step. Ent automatic schema creation is allowed only in disposable development/test contexts if the later implementation spec permits it; it must never run as part of production application startup. + +## Consequences + +- Domain-oriented code gains typed entities, edges, predicates, and compile-time feedback. +- Engineers must still inspect generated SQL, query plans, indexes, transaction boundaries, and possible N+1 behavior; the ORM is not a performance or correctness boundary. +- The importer owns a deliberate lower-level pgx path and cannot reuse ordinary Ent writes for convenience. +- Both Ent and pgx require compatible pool/transaction ownership and context propagation; adapters must not hide unbounded work. +- Migration generation tooling and the migration runner remain to be selected in the implementation spec. Versioned migration files—not runtime diffs—are the production contract. + +## References + +- Ent versioned migrations: +- pgx v5 `CopyFrom`: +