zarinpal-analytics-backend/docs/IMPORT-WORKFLOW-SPEC.md

One-Time CSV-to-PostgreSQL Import Workflow Specification

• Status: Approved specification — implementation not yet authorized
• Capability: Data foundation bootstrap
• Depends on: Approved PostgreSQL physical schema
• Implementation authorization: Not granted
• Last updated: 2026-08-20

Objective

Load the supplied challenge_data.csv.gz exactly once into the normalized PostgreSQL schema on the first server bootstrap, entirely through Docker. The workflow must be restart-safe, bounded-memory, observable, and unable to duplicate a previously completed dataset. It is not a recurring pipeline and does not run during ordinary application-container restarts.

Assumptions

1. The implementation uses a Bash entry script on the Linux server and the compiled Go importer from the backend image; Python is not added.
2. “On startup” means the explicit first-server bootstrap command, before the public application services are started, not every docker compose up or container restart.
3. The operator copies the archive into a Git-ignored deployment directory before running the first-start script.
4. The initial MVP database is empty of payment facts. Dataset replacement is a separate destructive recovery workflow and is not silently performed by this command.

Technology contract

• Orchestration: POSIX-compatible Bash with set -Eeuo pipefail
• Runtime: Docker Engine with Docker Compose v2
• Import process: compiled Go command from the backend repository
• PostgreSQL loading: pgx streaming decode plus PostgreSQL COPY into run-scoped staging
• Schema migrations: reviewed Atlas-generated SQL applied by a separate one-shot migration container before import
• Input: gzip-compressed UTF-8 CSV, mounted read-only
• Reports: JSON and Markdown files plus durable import_runs/import_issues rows

The Bash layer never parses CSV, constructs SQL, or contains normalization rules. Those rules remain versioned and tested with the Go importer.

Repository and server structure

zarinpal-analytics-backend/
  cmd/importer/                    # future Go composition root
  internal/importing/              # future stream/stage/validate/promote logic
  data/incoming/README.md          # tracked operator instructions
  docs/IMPORT-WORKFLOW-SPEC.md     # this contract

zarinpal-analytics-deployment/
  compose.yaml                     # future postgres/migrate/importer/runtime services
  scripts/first-start.sh           # future one-time orchestration entrypoint
  data/incoming/                   # Git-ignored source archive
    challenge_data.csv.gz
  data/reports/                    # Git-ignored generated reports

The deployment repository tracks scripts, Compose definitions, environment templates, and directory READMEs. The archive, generated reports, database volume, and secrets are ignored by Git.

Operator commands

The implementation must expose this first-server flow:

chmod +x ./scripts/first-start.sh
./scripts/first-start.sh

Equivalent explicit diagnostic commands are documented for operators:

docker compose up -d postgres redis
docker compose exec -T postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
docker compose --profile bootstrap run --rm migrate up
docker compose --profile bootstrap run --rm importer import \
  --source /import/input/challenge_data.csv.gz \
  --schema-version challenge-v1 \
  --report-json /import/reports/import-report.json \
  --report-md /import/reports/import-report.md
docker compose up -d backend worker frontend nginx

docker compose run --rm is used for the importer because it is a one-shot container, not a long-running service that can be addressed reliably with exec. docker compose exec -T is used for PostgreSQL readiness because PostgreSQL is already running.

Docker mount and service contract

The future importer Compose service:

• uses the same immutable backend image/revision as the API and worker;
• belongs to a bootstrap profile and has no restart policy;
• mounts ./data/incoming/challenge_data.csv.gz at /import/input/challenge_data.csv.gz:ro;
• mounts ./data/reports at /import/reports read-write;
• receives database credentials through the same private environment/secret mechanism as migrations;
• joins only the private application network;
• exposes no port;
• runs as a non-root user with a read-only root filesystem and a writable temporary directory where required;
• has bounded CPU/memory settings documented after profiling.

The archive is never copied into an image layer or uploaded through the public application.

First-start lifecycle

scripts/first-start.sh performs these steps in order:

1. Verify Docker Compose, required environment variables, source file presence/readability, and writable report directory.
2. Start PostgreSQL and Redis only. Wait for PostgreSQL readiness using bounded exponential backoff; fail with a clear non-secret error when the deadline expires.
3. Apply committed database migrations through the one-shot migration service. Application startup never performs schema changes.
4. Run importer preflight: safe basename, gzip integrity, exact supported header, source SHA-256, schema version, available disk-space check, and database connectivity.
5. If a completed import_runs row already exists for the same source checksum and schema version, write a no-op report and exit successfully without reading/promoting rows again.
6. Otherwise create an import run, stream the archive into run-scoped staging, validate, normalize, and promote according to the physical schema and validation specification.
7. Generate JSON and Markdown reports atomically through temporary files followed by rename. A failed or interrupted run remains non-completed and cannot identify the dataset snapshot.
8. Verify that exactly one completed import run exists and that reconciliation checks pass.
9. Start backend, worker, frontend, and Nginx only after successful completion or a verified idempotent no-op.

The script exits non-zero before public services start when migration, import, or reconciliation fails. It does not automatically delete staging evidence, drop the database, retry a different archive, or replace completed business data.

Import transaction and restart behavior

• Streaming/staging is bounded-memory and may commit staging batches associated with one import_run_id.
• Promotion to durable normalized tables is an explicit transaction or resumable versioned sequence whose final state transition is atomic.
• Only a run that passes all fatal checks may become completed and active.
• SIGINT, SIGTERM, container failure, PostgreSQL failure, or disk exhaustion records/retains a failed or interrupted non-active run when possible.
• Re-running after failure creates or resumes only through the approved recovery rules; it never assumes partially promoted rows are valid.
• Re-running after success with the same checksum/schema is a successful no-op.
• A different checksum fails closed with DATASET_REPLACEMENT_REQUIRES_EXPLICIT_RECOVERY; first-start never replaces the completed dataset.

Exact row rejection, staging cleanup, resume, and recovery rules are item 3 and remain gated separately.

Configuration contract

The implementation uses validated environment variables with safe defaults where appropriate:

Variable	Meaning
POSTGRES_USER, POSTGRES_DB, database secret	private database connection
IMPORT_SOURCE_PATH	container path; defaults to /import/input/challenge_data.csv.gz
IMPORT_SCHEMA_VERSION	required supported source contract, initially challenge-v1
IMPORT_REPORT_DIR	defaults to /import/reports
IMPORT_DB_READY_TIMEOUT_SECONDS	bounded first-start readiness deadline
IMPORT_COPY_BATCH_ROWS	validated bounded staging batch size

No environment variable may disable validation, tenant relationships, reconciliation, or sensitive-data redaction. Replacement/destructive flags are absent from the normal first-start interface.

Logging and report contract

Structured logs and reports may contain import-run id, checksum, schema/importer version, phase, duration, aggregate counts, stable rule codes, and bounded safe examples. They never contain:

• payer_card_key values;
• passwords, hashes, cookies, database credentials, or connection URLs;
• raw CSV rows;
• full operator filesystem paths;
• merchant/session/terminal source keys in routine logs.

Progress logging is rate-limited by time or aggregate row milestones rather than one line per row.

Testing strategy

• Unit tests: header parsing, gzip/preflight failures, timestamp conversion, normalization, no-attempt handling, redaction, and report encoding.
• PostgreSQL integration tests: staging, constraints, transaction boundaries, failed-run visibility, idempotent completed-run detection, and normalized reconciliation.
• Docker integration test: a small fixture executes the same first-start.sh/Compose profile and proves that a second invocation is a no-op.
• Interruption test: terminate the importer during staging and promotion and prove no incomplete run is selected as active.
• Production-shaped dry run: execute against the full 2,213,289-row archive and record duration, peak memory, disk growth, and final counts before deployment approval.

Success criteria

• One documented Bash command prepares dependencies, migrates, imports, verifies, and starts public services.
• The import runs entirely inside Docker and requires no host Go or Python installation.
• The archive remains read-only, outside Git, and outside container image layers.
• A clean database reaches exactly the approved normalized row counts and relationships.
• A second run with the identical source checksum/schema performs no promotion and exits successfully.
• A different archive cannot silently replace the completed dataset.
• Failure prevents public-service startup and produces safe actionable evidence.
• Ordinary docker compose up -d, service restart, deployment, and server reboot do not re-run the importer.

Boundaries

• Always: verify checksum/schema, use read-only input, apply committed migrations first, stage before promotion, reconcile counts, redact sensitive fields, and keep the import one-time/idempotent.
• Ask first: replace the dataset, resume or delete failed staging, change batch/resource limits, add Python, alter normalization, or run against a non-empty fact database.
• Never: run on a timer, import on every restart, expose an importer port, bake the dataset into an image, auto-migrate in application startup, log raw rows/tokens/secrets, or silently truncate production tables.

Resolved decisions

1. Use Bash for Docker orchestration and the compiled Go importer for CSV/database work; do not add Python.
2. Run import during the explicit first-start.sh bootstrap, not ordinary container startup or restart.
3. Mount data/incoming/challenge_data.csv.gz read-only and keep the archive/reports/database ignored by Git while tracking all scripts and documentation.
4. Treat a completed matching checksum/schema as a successful no-op; reject a different dataset unless an explicit recovery workflow is invoked.
5. Start public services only after migrations, import, and reconciliation succeed.