telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/design.md

## Context

See `proposal.md` for background and problem motivation.

The architecture is constrained to pure Python without web frameworks (FastAPI, Flask, Starlette, or Django). The pipeline must ingest high-throughput telecom CDRs, decouple ingestion from processing via Kafka, normalize heterogeneous fields, micro-batch into ClickHouse with at-least-once delivery guarantees, and provide complete operational observability via Prometheus metrics and structured JSON logging.

## Goals / Non-Goals

**Goals:**
- Implement a non-blocking HTTP ingestion server using Python standard library `asyncio` primitives.
- Enforce backpressure using bounded async queues (`429 Too Many Requests`).
- Publish to Kafka topic `cdr.raw` using `raw_id` partition keys with `confluent-kafka`.
- Build a resilient worker that normalizes phone numbers (E.164 via `phonenumbers`), maps RATs, generates deterministic UUIDv5 `cdr_id`s, and safely routes bad data to `cdr.raw.dlq`.
- Micro-batch inserts into ClickHouse (`ReplacingMergeTree`) with configurable count (1,000) and time (2s) flush thresholds.
- Guarantee at-least-once delivery by committing Kafka offsets only after ClickHouse write ACK.
- Expose dual-service Prometheus `/metrics` and aggregate public `/stats` via an internal worker HTTP endpoint.
- Emit structured JSON logs across all services.
- Containerize all services in `docker-compose.yml` with health checks and `condition: service_healthy`.

**Non-Goals:**
- Web-based visualization UI (analytical queries and verification scripts provide reporting).
- Schema Registry broker integration (deferred from optional bonus challenges).
- Distributed multi-worker consumer coordination (single worker instance fulfills baseline requirements).

## Decisions

### 1. Framework-less `asyncio` HTTP Server
- **Choice**: Implement the HTTP listener using `asyncio.start_server` with a lightweight, zero-dependency HTTP/1.1 request parser.
- **Rationale**: Strictly satisfies the constraint of "pure Python without frameworks" while delivering high concurrent connection handling and native integration with `asyncio.Queue` for backpressure.
- **Alternatives Considered**: 
  - `http.server.ThreadingHTTPServer`: Standard library, but thread-per-request model has higher memory overhead and makes bounded async backpressure coordination harder.
  - `FastAPI`/`Starlette`: Disallowed by project constraints.

### 2. Service Separation and Stats Aggregation
- **Choice**: Run `Ingestion API` and `Processing Worker` as separate Docker containers. The public `/stats` endpoint is hosted by the Ingestion API, which queries a lightweight internal HTTP endpoint on the Worker (`GET http://worker:8001/internal/stats`) and aggregates with local ingestion metrics.
- **Rationale**: Isolates HTTP ingestion from processing CPU/network bottlenecks. Enables independent scaling and lifecycle management.
- **Alternatives Considered**:
  - Single combined monolithic process: Violates explicit requirement for separate services.
  - Querying ClickHouse directly for `/stats`: Incurs database query overhead and does not report real-time in-flight buffer size or consumer lag.

### 3. Kafka Client and Partitioning Strategy
- **Choice**: Use `confluent-kafka` with `raw_id` as the message key when publishing to `cdr.raw`.
- **Rationale**: `confluent-kafka` is built on `librdkafka` and offers superior throughput over pure Python clients. Using `raw_id` ensures deterministic partition affinity for duplicate CDR submissions while spreading traffic evenly across the 3 partitions via Murmur2 hashing.
- **Alternatives Considered**:
  - `caller_num` as key: Risk of partition hotspots if certain subscribers generate disproportionate traffic.
  - `null` (round-robin): Loses record-level partition affinity.

### 4. ClickHouse Client and Idempotent Deduplication
- **Choice**: Use `clickhouse-connect` HTTP client targeting table `telecom.cdr_records` with engine `ReplacingMergeTree(processed_at)`.
- **Rationale**: ClickHouse recommends HTTP batch inserts. `ReplacingMergeTree` combined with deterministic UUIDv5 `cdr_id` ensures that if a batch is replayed after a worker crash prior to offset commit, duplicate records with identical order keys are safely merged without data duplication.
- **Alternatives Considered**:
  - `clickhouse-driver` (native TCP protocol): Slightly higher client-side setup complexity; `clickhouse-connect` is the official modern ClickHouse client.
  - Standard `MergeTree`: Does not deduplicate replayed batches, violating idempotency.

### 5. Micro-Batching and Fault Tolerance Strategy
- **Choice**: Worker accumulates records into an in-memory batch. Flushes when `len(batch) >= BATCH_SIZE` (default 1,000) or elapsed time >= `BATCH_FLUSH_INTERVAL_SECONDS` (default 2s).
- **Failure Handling**:
  - If ClickHouse write times out, worker retries with exponential backoff up to `CH_RETRY_MAX_ATTEMPTS` (default 3, initial delay 500ms, factor 2.0).
  - If all retries fail, the entire batch is wrapped and routed to `cdr.raw.dlq`, and offsets are committed to prevent pipeline stall.
  - On `SIGTERM`, consumer stops, remaining buffer is flushed to ClickHouse, offsets are committed, and worker exits cleanly.

### 6. Structured JSON Logging
- **Choice**: Standard library Python logging with a custom JSON formatter producing fields: `timestamp`, `level`, `service`, `raw_id`, and `event`.
- **Rationale**: Clean, zero-dependency implementation producing machine-readable logs compatible with modern observability stacks.

## Risks / Trade-offs

- **[Risk] High ingestion bursts saturate in-memory queue** → **[Mitigation]** The bounded `asyncio.Queue` rejects requests with `429 Too Many Requests` as soon as capacity is reached, protecting process memory.
- **[Risk] ClickHouse outage stalls worker** → **[Mitigation]** Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed to prevent infinite consumer lag loops.
- **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`.
- **[Risk] Phone number parsing performance overhead** → **[Mitigation]** Use `phonenumbers` library with fast-path prefix checks (`00`, `011`, `+`) before full parsing.