Network Element
    ↓
FastAPI API
    ↓
Validation
    ↓
Bounded Async Queue
    ↓
Redpanda / cdr.raw
    ↓
Processing Worker
    ↓
Validation + Normalization
    ├── invalid → cdr.raw.dlq
    └── valid
          ↓
      Micro-Batch
          ↓
      ClickHouse

Infrastructure already available:

Redpanda
├── cdr.raw
└── cdr.raw.dlq

ClickHouse
└── telecom.cdr_records

worker processing is not connected yet.

Current Features

Dockerized Redpanda and ClickHouse
Automatic and idempotent Kafka topic provisioning
Automatic ClickHouse schema initialization
Persistent Redpanda and ClickHouse volumes
Raw and normalized CDR domain models
Required-field and type validation
RAT validation and normalization
E.164 phone-number normalization
Deterministic UUIDv5 cdr_id generation
Epoch and traffic-value validation
FastAPI POST /api/v1/cdr/raw
FastAPI GET /health
Structured JSON error responses
Unit and API integration tests
Bounded asynchronous ingestion queue for backpressure
Long-lived asynchronous Kafka producer
Kafka delivery acknowledgement before 202 Accepted
429 Too Many Requests on ingestion queue saturation
503 Service Unavailable on Kafka delivery failure
Kafka consumer with manual offset management
Worker-side JSON decoding and validation
Worker-side CDR normalization
DLQ routing for malformed and unprocessable Kafka records
DLQ records with failure reason, failed field, timestamp, and worker instance
Kafka offsets committed only after the current processing outcome succeeds
ClickHouse bulk storage through the official Python client
Configurable CDR micro-batching
Flush on 1,000 records or 2-second timeout
Kafka offsets committed only after successful ClickHouse batch writes
Remaining worker buffer flushed during graceful shutdown

Project Structure

telecom-cdr-pipeline/
├── infrastructure/
│   └── clickhouse/
│       └── init.sql
│
├── scripts/
│   ├── sample-valid-cdr.json
│   └── sample-invalid-rat.json
│
├── src/
│   └── core/
│       ├── __init__.py
│       ├── config.py
│       │
│       ├── api/
│       │   ├── __init__.py
│       │   ├── kafka_publisher.py
│       │   ├── main.py
│       │   ├── publisher.py
│       │   └── routes.py
│       │
│       ├── domain/
│       │   ├── __init__.py
│       │   ├── errors.py
│       │   ├── models.py
│       │   ├── normalization.py
│       │   └── validation.py
│       │
│       └── worker/
│           ├── __init__.py
│           ├── batch.py
│           ├── clickhouse.py
│           ├── consumer.py
│           ├── dlq.py
│           ├── main.py
│           └── processor.py
│
├── tests/
│   ├── integration/
│   │   └── test_ingestion_api.py
│   │
│   └── unit/
│       ├── test_batch.py
│       ├── test_dlq.py
│       ├── test_normalization.py
│       ├── test_project_setup.py
│       ├── test_validation.py
│       └── test_worker_processor.py
│
├── .env.example
├── .gitignore
├── docker-compose.yml
├── pyproject.toml
└── README.md

Configuration

Configuration is provided through environment variables documented in .env.example.

This includes:

Redpanda version and topic settings
Kafka broker and consumer configuration
ClickHouse connection settings
Worker batch size and flush interval

Running Infrastructure

docker compose up -d

Verify services:

docker compose ps -a

Expected:

Redpanda       healthy
redpanda-init  exited successfully
ClickHouse     healthy

Running the API

uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000

Available endpoints:

GET  /health
POST /api/v1/cdr/raw

A valid CDR currently returns:

{
  "status": "accepted",
  "raw_id": "raw-test-001"
}

Invalid payloads return structured 400 responses.

Testing

Run:

pytest -v

The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.