Telecom CDR Pipeline

High-throughput telecom Call Detail Record ingestion and analytics pipeline.

Architecture

The system consists of:

FastAPI ingestion API
Redpanda as a Kafka-compatible message broker
Processing and normalization worker
ClickHouse analytical storage
Dead Letter Queue for failed records

Current data flow:

Network Element
    ↓
FastAPI API
    ↓
Validation
    ↓
Bounded Async Queue
    ↓
Redpanda / cdr.raw

Infrastructure already available:

Redpanda
├── cdr.raw
└── cdr.raw.dlq

ClickHouse
└── telecom.cdr_records

worker processing is not connected yet.

Current Features

Dockerized Redpanda and ClickHouse
Automatic and idempotent Kafka topic provisioning
Automatic ClickHouse schema initialization
Persistent Redpanda and ClickHouse volumes
Raw and normalized CDR domain models
Required-field and type validation
RAT validation and normalization
E.164 phone-number normalization
Deterministic UUIDv5 cdr_id generation
Epoch and traffic-value validation
FastAPI POST /api/v1/cdr/raw
FastAPI GET /health
Structured JSON error responses
Unit and API integration tests
Bounded asynchronous ingestion queue for backpressure
Long-lived asynchronous Kafka producer
Kafka delivery acknowledgement before 202 Accepted
429 Too Many Requests on ingestion queue saturation
503 Service Unavailable on Kafka delivery failure

Project Structure

src/core/
├── config.py
│
├── api/
│   ├── __init__.py
│   ├── kafka_publisher.py
│   ├── main.py
│   ├── publisher.py
│   └── routes.py
│
└── domain/
    ├── __init__.py
    ├── errors.py
    ├── models.py
    ├── normalization.py
    └── validation.py

infrastructure/
└── clickhouse/
    └── init.sql

tests/
├── integration/
└── unit/

Configuration

Configuration is provided through environment variables documented in .env.example.

This includes:

Redpanda version and topic settings
Kafka broker and consumer configuration
ClickHouse connection settings
Worker batch size and flush interval

Running Infrastructure

docker compose up -d

Verify services:

docker compose ps -a

Expected:

Redpanda       healthy
redpanda-init  exited successfully
ClickHouse     healthy

Running the API

uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000

Available endpoints:

GET  /health
POST /api/v1/cdr/raw

A valid CDR currently returns:

{
  "status": "accepted",
  "raw_id": "raw-test-001"
}

Invalid payloads return structured 400 responses.

Testing

Run:

pytest -v

The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.

2.9 KiB Raw Blame History