179 lines
11 KiB
Markdown
179 lines
11 KiB
Markdown
# One-Time CSV-to-PostgreSQL Import Workflow Specification
|
|
|
|
- Status: Approved specification — implementation authorized 2026-08-20
|
|
- Capability: Data foundation bootstrap
|
|
- Depends on: Approved PostgreSQL physical schema
|
|
- Implementation authorization: Granted by user on 2026-08-20
|
|
- Last updated: 2026-08-20
|
|
|
|
## Objective
|
|
|
|
Load the supplied `challenge_data.csv.gz` exactly once into the normalized PostgreSQL schema on the first server bootstrap, entirely through Docker. The workflow must be restart-safe, bounded-memory, observable, and unable to duplicate a previously completed dataset. It is not a recurring pipeline and does not run during ordinary application-container restarts.
|
|
|
|
## Assumptions
|
|
|
|
1. The implementation uses a Bash entry script on the Linux server and the compiled Go importer from the backend image; Python is not added.
|
|
2. “On startup” means the explicit first-server bootstrap command, before the public application services are started, not every `docker compose up` or container restart.
|
|
3. The operator copies the archive into a Git-ignored deployment directory before running the first-start script.
|
|
4. The initial MVP database is empty of payment facts. Dataset replacement is a separate destructive recovery workflow and is not silently performed by this command.
|
|
|
|
## Technology contract
|
|
|
|
- Orchestration: POSIX-compatible Bash with `set -Eeuo pipefail`
|
|
- Runtime: Docker Engine with Docker Compose v2
|
|
- Import process: compiled Go command from the backend repository
|
|
- PostgreSQL loading: pgx streaming decode plus PostgreSQL `COPY` into run-scoped staging
|
|
- Schema migrations: reviewed Atlas-generated SQL applied by a separate one-shot migration container before import
|
|
- Input: gzip-compressed UTF-8 CSV, mounted read-only
|
|
- Reports: JSON and Markdown files plus durable `import_runs`/`import_issues` rows
|
|
|
|
The Bash layer never parses CSV, constructs SQL, or contains normalization rules. Those rules remain versioned and tested with the Go importer.
|
|
|
|
## Repository and server structure
|
|
|
|
```text
|
|
zarinpal-analytics-backend/
|
|
cmd/importer/ # future Go composition root
|
|
internal/importing/ # future stream/stage/validate/promote logic
|
|
data/incoming/README.md # tracked operator instructions
|
|
docs/IMPORT-WORKFLOW-SPEC.md # this contract
|
|
|
|
zarinpal-analytics-deployment/
|
|
compose.yaml # future postgres/migrate/importer/runtime services
|
|
scripts/first-start.sh # future one-time orchestration entrypoint
|
|
data/incoming/ # Git-ignored source archive
|
|
challenge_data.csv.gz
|
|
data/reports/ # Git-ignored generated reports
|
|
```
|
|
|
|
The deployment repository tracks scripts, Compose definitions, environment templates, and directory READMEs. The archive, generated reports, database volume, and secrets are ignored by Git.
|
|
|
|
## Operator commands
|
|
|
|
The implementation must expose this first-server flow:
|
|
|
|
```bash
|
|
chmod +x ./scripts/first-start.sh
|
|
./scripts/first-start.sh
|
|
```
|
|
|
|
Equivalent explicit diagnostic commands are documented for operators:
|
|
|
|
```bash
|
|
docker compose up -d postgres redis
|
|
docker compose exec -T postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
|
|
docker compose --profile bootstrap run --rm migrate up
|
|
docker compose --profile bootstrap run --rm importer import \
|
|
--source /import/input/challenge_data.csv.gz \
|
|
--schema-version challenge-v1 \
|
|
--report-json /import/reports/import-report.json \
|
|
--report-md /import/reports/import-report.md
|
|
docker compose up -d backend worker frontend nginx
|
|
```
|
|
|
|
`docker compose run --rm` is used for the importer because it is a one-shot container, not a long-running service that can be addressed reliably with `exec`. `docker compose exec -T` is used for PostgreSQL readiness because PostgreSQL is already running.
|
|
|
|
## Docker mount and service contract
|
|
|
|
The future `importer` Compose service:
|
|
|
|
- uses the same immutable backend image/revision as the API and worker;
|
|
- belongs to a `bootstrap` profile and has no restart policy;
|
|
- mounts `./data/incoming/challenge_data.csv.gz` at `/import/input/challenge_data.csv.gz:ro`;
|
|
- mounts `./data/reports` at `/import/reports` read-write;
|
|
- receives database credentials through the same private environment/secret mechanism as migrations;
|
|
- joins only the private application network;
|
|
- exposes no port;
|
|
- runs as a non-root user with a read-only root filesystem and a writable temporary directory where required;
|
|
- has bounded CPU/memory settings documented after profiling.
|
|
|
|
The archive is never copied into an image layer or uploaded through the public application.
|
|
|
|
## First-start lifecycle
|
|
|
|
`scripts/first-start.sh` performs these steps in order:
|
|
|
|
1. Verify Docker Compose, required environment variables, source file presence/readability, and writable report directory.
|
|
2. Start PostgreSQL and Redis only. Wait for PostgreSQL readiness using bounded exponential backoff; fail with a clear non-secret error when the deadline expires.
|
|
3. Apply committed database migrations through the one-shot migration service. Application startup never performs schema changes.
|
|
4. Run importer preflight: safe basename, gzip integrity, exact supported header, source SHA-256, schema version, available disk-space check, and database connectivity.
|
|
5. If a completed `import_runs` row already exists for the same source checksum and schema version, write a no-op report and exit successfully without reading/promoting rows again.
|
|
6. Otherwise create an import run, stream the archive into run-scoped staging, validate, normalize, and promote according to the physical schema and validation specification.
|
|
7. Generate JSON and Markdown reports atomically through temporary files followed by rename. A failed or interrupted run remains non-completed and cannot identify the dataset snapshot.
|
|
8. Verify that exactly one completed import run exists and that reconciliation checks pass.
|
|
9. Start backend, worker, frontend, and Nginx only after successful completion or a verified idempotent no-op.
|
|
|
|
The script exits non-zero before public services start when migration, import, or reconciliation fails. It does not automatically delete staging evidence, drop the database, retry a different archive, or replace completed business data.
|
|
|
|
## Import transaction and restart behavior
|
|
|
|
- Streaming/staging is bounded-memory and may commit staging batches associated with one `import_run_id`.
|
|
- Promotion to durable normalized tables is an explicit transaction or resumable versioned sequence whose final state transition is atomic.
|
|
- Only a run that passes all fatal checks may become `completed` and active.
|
|
- SIGINT, SIGTERM, container failure, PostgreSQL failure, or disk exhaustion records/retains a failed or interrupted non-active run when possible.
|
|
- Re-running after failure creates or resumes only through the approved recovery rules; it never assumes partially promoted rows are valid.
|
|
- Re-running after success with the same checksum/schema is a successful no-op.
|
|
- A different checksum fails closed with `DATASET_REPLACEMENT_REQUIRES_EXPLICIT_RECOVERY`; first-start never replaces the completed dataset.
|
|
|
|
Exact row rejection, staging cleanup, interruption, and recovery rules are defined in [`IMPORT-VALIDATION-RECOVERY-SPEC.md`](./IMPORT-VALIDATION-RECOVERY-SPEC.md) and remain gated separately until stakeholder approval.
|
|
|
|
## Configuration contract
|
|
|
|
The implementation uses validated environment variables with safe defaults where appropriate:
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `POSTGRES_USER`, `POSTGRES_DB`, database secret | private database connection |
|
|
| `IMPORT_SOURCE_PATH` | container path; defaults to `/import/input/challenge_data.csv.gz` |
|
|
| `IMPORT_SCHEMA_VERSION` | required supported source contract, initially `challenge-v1` |
|
|
| `IMPORT_REPORT_DIR` | defaults to `/import/reports` |
|
|
| `IMPORT_DB_READY_TIMEOUT_SECONDS` | bounded first-start readiness deadline |
|
|
| `IMPORT_COPY_BATCH_ROWS` | validated bounded staging batch size |
|
|
|
|
No environment variable may disable validation, tenant relationships, reconciliation, or sensitive-data redaction. Replacement/destructive flags are absent from the normal first-start interface.
|
|
|
|
## Logging and report contract
|
|
|
|
Structured logs and reports may contain import-run id, checksum, schema/importer version, phase, duration, aggregate counts, stable rule codes, and bounded safe examples. They never contain:
|
|
|
|
- `payer_card_key` values;
|
|
- passwords, hashes, cookies, database credentials, or connection URLs;
|
|
- raw CSV rows;
|
|
- full operator filesystem paths;
|
|
- merchant/session/terminal source keys in routine logs.
|
|
|
|
Progress logging is rate-limited by time or aggregate row milestones rather than one line per row.
|
|
|
|
## Testing strategy
|
|
|
|
- Unit tests: header parsing, gzip/preflight failures, timestamp conversion, normalization, no-attempt handling, redaction, and report encoding.
|
|
- PostgreSQL integration tests: staging, constraints, transaction boundaries, failed-run visibility, idempotent completed-run detection, and normalized reconciliation.
|
|
- Docker integration test: a small fixture executes the same `first-start.sh`/Compose profile and proves that a second invocation is a no-op.
|
|
- Interruption test: terminate the importer during staging and promotion and prove no incomplete run is selected as active.
|
|
- Production-shaped dry run: execute against the full 2,213,289-row archive and record duration, peak memory, disk growth, and final counts before deployment approval.
|
|
|
|
## Success criteria
|
|
|
|
- One documented Bash command prepares dependencies, migrates, imports, verifies, and starts public services.
|
|
- The import runs entirely inside Docker and requires no host Go or Python installation.
|
|
- The archive remains read-only, outside Git, and outside container image layers.
|
|
- A clean database reaches exactly the approved normalized row counts and relationships.
|
|
- A second run with the identical source checksum/schema performs no promotion and exits successfully.
|
|
- A different archive cannot silently replace the completed dataset.
|
|
- Failure prevents public-service startup and produces safe actionable evidence.
|
|
- Ordinary `docker compose up -d`, service restart, deployment, and server reboot do not re-run the importer.
|
|
|
|
## Boundaries
|
|
|
|
- Always: verify checksum/schema, use read-only input, apply committed migrations first, stage before promotion, reconcile counts, redact sensitive fields, and keep the import one-time/idempotent.
|
|
- Ask first: replace the dataset, resume or delete failed staging, change batch/resource limits, add Python, alter normalization, or run against a non-empty fact database.
|
|
- Never: run on a timer, import on every restart, expose an importer port, bake the dataset into an image, auto-migrate in application startup, log raw rows/tokens/secrets, or silently truncate production tables.
|
|
|
|
## Resolved decisions
|
|
|
|
1. Use Bash for Docker orchestration and the compiled Go importer for CSV/database work; do not add Python.
|
|
2. Run import during the explicit `first-start.sh` bootstrap, not ordinary container startup or restart.
|
|
3. Mount `data/incoming/challenge_data.csv.gz` read-only and keep the archive/reports/database ignored by Git while tracking all scripts and documentation.
|
|
4. Treat a completed matching checksum/schema as a successful no-op; reject a different dataset unless an explicit recovery workflow is invoked.
|
|
5. Start public services only after migrations, import, and reconciliation succeed.
|