telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/proposal.md

Why

Telecom network elements generate high-volume, heterogeneous Call Detail Records (CDRs) that must be ingested, validated, normalized, and stored in near real-time without data loss. This change establishes a resilient, high-throughput ingestion and analytics pipeline using pure Python (framework-less asyncio), Kafka, and ClickHouse to guarantee at-least-once delivery, backpressure protection, deterministic deduplication, and full observability.

What Changes

• Ingestion API: A framework-less asyncio HTTP service accepting raw CDRs at POST /api/v1/cdr/raw, applying bounded queue backpressure (429), publishing to Kafka (cdr.raw), and serving /metrics and aggregated /stats.
• Message Broker & DLQ: Kafka topic definitions for cdr.raw (3 partitions, keyed by raw_id) and cdr.raw.dlq (1 partition) with a standardized error envelope schema.
• Processing Worker: A resilient consumer worker performing schema validation, E.164 normalization, RAT mapping, deterministic UUIDv5 ID generation, micro-batching (1,000 records or 2 seconds), and offset commits only after ClickHouse ACK.
• ClickHouse Storage: ReplacingMergeTree table schema in telecom.cdr_records with automated idempotent schema initialization, toIPv4() casting, and 90-day TTL.
• Observability & Logging: Structured JSON logging (timestamp, level, service, raw_id, event), Prometheus metrics (counters, histograms, gauges), and dual-service stats aggregation.
• Verification & Deployment: Docker Compose orchestrating all services with health checks and condition: service_healthy, alongside an automated traffic generator script verifying all 7 test scenarios.

Capabilities

New Capabilities

• ingestion-api: Framework-less asyncio HTTP gate, bounded queue backpressure, Kafka producer, /metrics, and aggregated /stats.
• message-broker: Topic definitions, partition strategy with raw_id, and DLQ envelope schema with failure taxonomy.
• processing-worker: Consumer loop, data cleansing, E.164 normalization, RAT mapping, micro-batching, at-least-once offset management, and graceful shutdown.
• clickhouse-storage: Database DDL, ReplacingMergeTree table design, toIPv4() casting, and idempotent initialization.
• observability: Structured JSON logging, Prometheus metrics definitions, and internal/public stats reporting.
• verification: Automated traffic generator covering 7 scenarios and pipeline verification reporting.

Modified Capabilities

Impact

• New Services: Ingestion API service and Processing Worker service.
• Dependencies: confluent-kafka, clickhouse-connect, phonenumbers (no web frameworks).
• Infrastructure: Kafka/Redpanda broker and ClickHouse database managed via Docker Compose.
• Data Flow: Network Element → HTTP API → Kafka (cdr.raw) → Worker → ClickHouse (cdr_records) / DLQ (cdr.raw.dlq).

Delivery safety correction

Broker acknowledgement defines API acceptance. Explicit partition offsets advance only after acknowledged ClickHouse or DLQ storage. The assessment SQL omits identity from its replacement key; append cdr_id to prevent distinct events collapsing. Required-field HTTP rejection and direct Kafka fault injection resolve contradictory assessment examples. Existing assessment data is never reset automatically.