telecom-cdr-pipeline/README.md

Telecom CDR Pipeline

High-throughput telecom Call Detail Record ingestion and analytics pipeline.

Architecture

The system consists of:

• FastAPI ingestion API
• Redpanda as a Kafka-compatible message broker
• Processing and normalization worker
• ClickHouse analytical storage
• Dead Letter Queue for failed records

Current data flow:

Network Element
    ↓
FastAPI API
    ↓
Validation
    ↓
Bounded Async Queue
    ↓
Redpanda / cdr.raw
    ↓
Processing Worker
    ↓
Validation + Normalization
    ├── invalid → cdr.raw.dlq
    └── valid
          ↓
      Micro-Batch
          ↓
      ClickHouse

Infrastructure already available:

Redpanda
├── cdr.raw
└── cdr.raw.dlq

ClickHouse
└── telecom.cdr_records

worker processing is not connected yet.

Current Features

• Dockerized Redpanda and ClickHouse
• Automatic and idempotent Kafka topic provisioning
• Automatic ClickHouse schema initialization
• Persistent Redpanda and ClickHouse volumes
• Raw and normalized CDR domain models
• Required-field and type validation
• RAT validation and normalization
• E.164 phone-number normalization
• Deterministic UUIDv5 cdr_id generation
• Epoch and traffic-value validation
• FastAPI POST /api/v1/cdr/raw
• FastAPI GET /health
• Structured JSON error responses
• Unit and API integration tests
• Bounded asynchronous ingestion queue for backpressure
• Long-lived asynchronous Kafka producer
• Kafka delivery acknowledgement before 202 Accepted
• 429 Too Many Requests on ingestion queue saturation
• 503 Service Unavailable on Kafka delivery failure
• Kafka consumer with manual offset management
• Worker-side JSON decoding and validation
• Worker-side CDR normalization
• DLQ routing for malformed and unprocessable Kafka records
• DLQ records with failure reason, failed field, timestamp, and worker instance
• Kafka offsets committed only after the current processing outcome succeeds
• ClickHouse bulk storage through the official Python client
• Configurable CDR micro-batching
• Flush on 1,000 records or 2-second timeout
• Kafka offsets committed only after successful ClickHouse batch writes
• Remaining worker buffer flushed during graceful shutdown
• ClickHouse insert retries with exponential backoff
• Failed ClickHouse batches routed to the DLQ after three attempts
• Whole-batch offset commits with no partial Kafka acknowledgement
• Replay-safe storage using deterministic UUIDv5 identifiers and ReplacingMergeTree
• Graceful worker shutdown flushes buffered records before exit
• Separate Prometheus /metrics endpoints for the API and worker
• API accepted-CDR counter
• Worker processed and DLQ counters
• Kafka-to-ClickHouse and batch flush latency histograms
• Batch size and in-flight buffer metrics
• Kafka consumer lag by partition
• Aggregated human-readable /stats endpoint
• Fully containerized API and processing worker
• Health-based Docker Compose startup ordering
• Automatic Kafka topic and ClickHouse schema provisioning
• Persistent Redpanda and ClickHouse storage
• Complete stack startup with docker compose up --build

Project Structure

telecom-cdr-pipeline/
├── infrastructure/
│   └── clickhouse/
│       └── init.sql
│
├── scripts/
│   ├── sample-valid-cdr.json
│   └── sample-invalid-rat.json
│
├── src/
│   └── core/
│       ├── __init__.py
│       ├── config.py
│       │
│       ├── api/
│       │   ├── __init__.py
│       │   ├── kafka_publisher.py
│       │   ├── main.py
│       │   ├── metrics.py
│       │   ├── publisher.py
│       │   ├── routes.py
│       │   └── stats.py
│       │
│       ├── domain/
│       │   ├── __init__.py
│       │   ├── errors.py
│       │   ├── models.py
│       │   ├── normalization.py
│       │   └── validation.py
│       │
│       └── worker/
│           ├── __init__.py
│           ├── batch.py
│           ├── clickhouse.py
│           ├── consumer.py
│           ├── dlq.py
│           ├── main.py
│           ├── metrics.py
│           ├── observability.py
│           ├── processor.py
│           └── stats.py
│
├── tests/
│   ├── integration/
│   │   └── test_ingestion_api.py
│   │
│   └── unit/
│       ├── test_batch.py
│       ├── test_dlq.py
│       ├── test_normalization.py
│       ├── test_project_setup.py
│       ├── test_validation.py
│       ├── test_worker_processor.py
│       └── test_worker_stats.py.py
│
├── .env.example
├── .gitignore
├── docker-compose.yml
├── pyproject.toml
└── README.md

Configuration

Configuration is provided through environment variables documented in .env.example.

This includes:

• Redpanda version and topic settings
• Kafka broker and consumer configuration
• ClickHouse connection settings
• Worker batch size and flush interval

Running Pipeline

docker compose up -d --build

Verify services:

docker compose ps -a

Expected:

Redpanda       healthy
redpanda-init  exited successfully
ClickHouse     healthy

Available endpoints:

API:

GET  http://localhost:8000/health
GET  http://localhost:8000/metrics
GET  http://localhost:8000/stats
POST http://localhost:8000/api/v1/cdr/raw

Worker:

GET  http://localhost:8001/health
GET  http://localhost:8001/metrics
GET  http://localhost:8001/internal/stats

A valid CDR currently returns:

{
  "status": "accepted",
  "raw_id": "raw-test-001"
}

Invalid payloads return structured 400 responses.

Testing

Once the full stack starts cleanly, a real CDR reaches ClickHouse, and:

pytest -v

Delivery Semantics

The worker uses at-least-once delivery.

Kafka offsets are committed only after a ClickHouse batch is successfully written or, after exhausted retries, the complete failed batch is acknowledged by the DLQ.

Repeated records use deterministic UUIDv5 identifiers and ReplacingMergeTree to support replay-safe storage.