telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/design.md

Context

See proposal.md for background and problem motivation.

The architecture is constrained to pure Python without web frameworks (FastAPI, Flask, Starlette, or Django). The pipeline must ingest high-throughput telecom CDRs, decouple ingestion from processing via Kafka, normalize heterogeneous fields, micro-batch into ClickHouse with at-least-once delivery guarantees, and provide complete operational observability via Prometheus metrics and structured JSON logging.

Goals / Non-Goals

Goals:

• Implement a non-blocking HTTP ingestion server using Python standard library asyncio primitives.
• Enforce backpressure using bounded async queues (429 Too Many Requests).
• Publish to Kafka topic cdr.raw using raw_id partition keys with confluent-kafka.
• Build a resilient worker that normalizes phone numbers (E.164 via phonenumbers), maps RATs, generates deterministic UUIDv5 cdr_ids, and safely routes bad data to cdr.raw.dlq.
• Micro-batch inserts into ClickHouse (ReplacingMergeTree) with configurable count (1,000) and time (2s) flush thresholds.
• Guarantee at-least-once delivery by committing Kafka offsets only after ClickHouse write ACK.
• Expose dual-service Prometheus /metrics and aggregate public /stats via an internal worker HTTP endpoint.
• Emit structured JSON logs across all services.
• Containerize all services in docker-compose.yml with health checks and condition: service_healthy.

Non-Goals:

• Web-based visualization UI (analytical queries and verification scripts provide reporting).
• Schema Registry broker integration (deferred from optional bonus challenges).
• Distributed multi-worker consumer coordination (single worker instance fulfills baseline requirements).

Decisions

1. Framework-less asyncio HTTP Server

• Choice: Implement the HTTP listener using asyncio.start_server with a lightweight, zero-dependency HTTP/1.1 request parser.
• Rationale: Strictly satisfies the constraint of "pure Python without frameworks" while delivering high concurrent connection handling and native integration with asyncio.Queue for backpressure.
• Alternatives Considered:
  • http.server.ThreadingHTTPServer: Standard library, but thread-per-request model has higher memory overhead and makes bounded async backpressure coordination harder.
  • FastAPI/Starlette: Disallowed by project constraints.

2. Service Separation and Stats Aggregation

• Choice: Run Ingestion API and Processing Worker as separate Docker containers. The public /stats endpoint is hosted by the Ingestion API, which queries a lightweight internal HTTP endpoint on the Worker (GET http://worker:8001/internal/stats) and aggregates with local ingestion metrics.
• Rationale: Isolates HTTP ingestion from processing CPU/network bottlenecks. Enables independent scaling and lifecycle management.
• Alternatives Considered:
  • Single combined monolithic process: Violates explicit requirement for separate services.
  • Querying ClickHouse directly for /stats: Incurs database query overhead and does not report real-time in-flight buffer size or consumer lag.

3. Kafka Client and Partitioning Strategy

• Choice: Use confluent-kafka with raw_id as the message key when publishing to cdr.raw.
• Rationale: confluent-kafka is built on librdkafka and offers superior throughput over pure Python clients. Using raw_id ensures deterministic partition affinity for duplicate CDR submissions while spreading traffic evenly across the 3 partitions via Murmur2 hashing.
• Alternatives Considered:
  • caller_num as key: Risk of partition hotspots if certain subscribers generate disproportionate traffic.
  • null (round-robin): Loses record-level partition affinity.

4. ClickHouse Client and Idempotent Deduplication

• Choice: Use clickhouse-connect HTTP client targeting table telecom.cdr_records with engine ReplacingMergeTree(processed_at).
• Rationale: ClickHouse recommends HTTP batch inserts. ReplacingMergeTree combined with deterministic UUIDv5 cdr_id ensures that if a batch is replayed after a worker crash prior to offset commit, duplicate records with identical order keys are safely merged without data duplication.
• Alternatives Considered:
  • clickhouse-driver (native TCP protocol): Slightly higher client-side setup complexity; clickhouse-connect is the official modern ClickHouse client.
  • Standard MergeTree: Does not deduplicate replayed batches, violating idempotency.

5. Micro-Batching and Fault Tolerance Strategy

• Choice: Worker accumulates records into an in-memory batch. Flushes when len(batch) >= BATCH_SIZE (default 1,000) or elapsed time >= BATCH_FLUSH_INTERVAL_SECONDS (default 2s).
• Failure Handling:
  • If ClickHouse write times out, worker retries with exponential backoff up to CH_RETRY_MAX_ATTEMPTS (default 3, initial delay 500ms, factor 2.0).
  • If all retries fail, the entire batch is wrapped and routed to cdr.raw.dlq, and offsets are committed to prevent pipeline stall.
  • On SIGTERM, consumer stops, remaining buffer is flushed to ClickHouse, offsets are committed, and worker exits cleanly.

6. Structured JSON Logging

• Choice: Standard library Python logging with a custom JSON formatter producing fields: timestamp, level, service, raw_id, and event.
• Rationale: Clean, zero-dependency implementation producing machine-readable logs compatible with modern observability stacks.

Risks / Trade-offs

• [Risk] High ingestion bursts saturate in-memory queue → [Mitigation] The bounded asyncio.Queue rejects requests with 429 Too Many Requests as soon as capacity is reached, protecting process memory.
• [Risk] ClickHouse outage stalls worker → [Mitigation] Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed to prevent infinite consumer lag loops.
• [Risk] Worker crashes between ClickHouse ACK and Kafka commit → [Mitigation] Re-consumed batch is re-inserted; ReplacingMergeTree(processed_at) deduplicates rows sharing the order key (service_type, radio_access_type, cell_id, session_start_time, msisdn).
• [Risk] Phone number parsing performance overhead → [Mitigation] Use phonenumbers library with fast-path prefix checks (00, 011, +) before full parsing.