36 lines
3.3 KiB
Markdown
36 lines
3.3 KiB
Markdown
## Why
|
|
|
|
Telecom network elements generate high-volume, heterogeneous Call Detail Records (CDRs) that must be ingested, validated, normalized, and stored in near real-time without data loss. This change establishes a resilient, high-throughput ingestion and analytics pipeline using pure Python (framework-less `asyncio`), Kafka, and ClickHouse to guarantee at-least-once delivery, backpressure protection, deterministic deduplication, and full observability.
|
|
|
|
## What Changes
|
|
|
|
- **Ingestion API**: A framework-less `asyncio` HTTP service accepting raw CDRs at `POST /api/v1/cdr/raw`, applying bounded queue backpressure (429), publishing to Kafka (`cdr.raw`), and serving `/metrics` and aggregated `/stats`.
|
|
- **Message Broker & DLQ**: Kafka topic definitions for `cdr.raw` (3 partitions, keyed by `raw_id`) and `cdr.raw.dlq` (1 partition) with a standardized error envelope schema.
|
|
- **Processing Worker**: A resilient consumer worker performing schema validation, E.164 normalization, RAT mapping, deterministic UUIDv5 ID generation, micro-batching (1,000 records or 2 seconds), and offset commits only after ClickHouse ACK.
|
|
- **ClickHouse Storage**: `ReplacingMergeTree` table schema in `telecom.cdr_records` with automated idempotent schema initialization, `toIPv4()` casting, and 90-day TTL.
|
|
- **Observability & Logging**: Structured JSON logging (`timestamp`, `level`, `service`, `raw_id`, `event`), Prometheus metrics (counters, histograms, gauges), and dual-service stats aggregation.
|
|
- **Verification & Deployment**: Docker Compose orchestrating all services with health checks and `condition: service_healthy`, alongside an automated traffic generator script verifying all 7 test scenarios.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
- `ingestion-api`: Framework-less `asyncio` HTTP gate, bounded queue backpressure, Kafka producer, `/metrics`, and aggregated `/stats`.
|
|
- `message-broker`: Topic definitions, partition strategy with `raw_id`, and DLQ envelope schema with failure taxonomy.
|
|
- `processing-worker`: Consumer loop, data cleansing, E.164 normalization, RAT mapping, micro-batching, at-least-once offset management, and graceful shutdown.
|
|
- `clickhouse-storage`: Database DDL, `ReplacingMergeTree` table design, `toIPv4()` casting, and idempotent initialization.
|
|
- `observability`: Structured JSON logging, Prometheus metrics definitions, and internal/public stats reporting.
|
|
- `verification`: Automated traffic generator covering 7 scenarios and pipeline verification reporting.
|
|
|
|
### Modified Capabilities
|
|
<!-- None: Initial implementation -->
|
|
|
|
## Impact
|
|
|
|
- **New Services**: Ingestion API service and Processing Worker service.
|
|
- **Dependencies**: `confluent-kafka`, `clickhouse-connect`, `phonenumbers` (no web frameworks).
|
|
- **Infrastructure**: Kafka/Redpanda broker and ClickHouse database managed via Docker Compose.
|
|
- **Data Flow**: Network Element → HTTP API → Kafka (`cdr.raw`) → Worker → ClickHouse (`cdr_records`) / DLQ (`cdr.raw.dlq`).
|
|
|
|
## Delivery safety correction
|
|
Broker acknowledgement defines API acceptance. Explicit partition offsets advance only after acknowledged ClickHouse or DLQ storage. The assessment SQL omits identity from its replacement key; append cdr_id to prevent distinct events collapsing. Required-field HTTP rejection and direct Kafka fault injection resolve contradictory assessment examples. Existing assessment data is never reset automatically.
|