telecom-cdr-pipeline/README.md

# Telecom CDR Pipeline

High-throughput telecom Call Detail Record ingestion and analytics pipeline.

## Architecture

The system consists of:

* FastAPI ingestion API
* Redpanda as a Kafka-compatible message broker
* Processing and normalization worker
* ClickHouse analytical storage
* Dead Letter Queue for failed records

Current data flow:

```text
Network Element
    ↓
FastAPI API
    ↓
Validation
    ↓
Bounded Async Queue
    ↓
Redpanda / cdr.raw
    ↓
Processing Worker
    ↓
Validation + Normalization
    ├── invalid → cdr.raw.dlq
    └── valid
          ↓
      Micro-Batch
          ↓
      ClickHouse
```

Infrastructure already available:

```text
Redpanda
├── cdr.raw
└── cdr.raw.dlq

ClickHouse
└── telecom.cdr_records
```

worker processing is not connected yet.

## Current Features

* Dockerized Redpanda and ClickHouse
* Automatic and idempotent Kafka topic provisioning
* Automatic ClickHouse schema initialization
* Persistent Redpanda and ClickHouse volumes
* Raw and normalized CDR domain models
* Required-field and type validation
* RAT validation and normalization
* E.164 phone-number normalization
* Deterministic UUIDv5 `cdr_id` generation
* Epoch and traffic-value validation
* FastAPI `POST /api/v1/cdr/raw`
* FastAPI `GET /health`
* Structured JSON error responses
* Unit and API integration tests
* Bounded asynchronous ingestion queue for backpressure
* Long-lived asynchronous Kafka producer
* Kafka delivery acknowledgement before `202 Accepted`
* `429 Too Many Requests` on ingestion queue saturation
* `503 Service Unavailable` on Kafka delivery failure
* Kafka consumer with manual offset management
* Worker-side JSON decoding and validation
* Worker-side CDR normalization
* DLQ routing for malformed and unprocessable Kafka records
* DLQ records with failure reason, failed field, timestamp, and worker instance
* Kafka offsets committed only after the current processing outcome succeeds
* ClickHouse bulk storage through the official Python client
* Configurable CDR micro-batching
* Flush on 1,000 records or 2-second timeout
* Kafka offsets committed only after successful ClickHouse batch writes
* Remaining worker buffer flushed during graceful shutdown
* ClickHouse insert retries with exponential backoff
* Failed ClickHouse batches routed to the DLQ after three attempts
* Whole-batch offset commits with no partial Kafka acknowledgement
* Replay-safe storage using deterministic UUIDv5 identifiers and `ReplacingMergeTree`
* Graceful worker shutdown flushes buffered records before exit
* Separate Prometheus `/metrics` endpoints for the API and worker
* API accepted-CDR counter
* Worker processed and DLQ counters
* Kafka-to-ClickHouse and batch flush latency histograms
* Batch size and in-flight buffer metrics
* Kafka consumer lag by partition
* Aggregated human-readable `/stats` endpoint
* Fully containerized API and processing worker
* Health-based Docker Compose startup ordering
* Automatic Kafka topic and ClickHouse schema provisioning
* Persistent Redpanda and ClickHouse storage
* Complete stack startup with `docker compose up --build`

## Project Structure

```text
telecom-cdr-pipeline/
├── infrastructure/
│   └── clickhouse/
│       └── init.sql
│
├── scripts/
│   ├── sample-valid-cdr.json
│   └── sample-invalid-rat.json
│
├── src/
│   └── core/
│       ├── __init__.py
│       ├── config.py
│       │
│       ├── api/
│       │   ├── __init__.py
│       │   ├── kafka_publisher.py
│       │   ├── main.py
│       │   ├── metrics.py
│       │   ├── publisher.py
│       │   ├── routes.py
│       │   └── stats.py
│       │
│       ├── domain/
│       │   ├── __init__.py
│       │   ├── errors.py
│       │   ├── models.py
│       │   ├── normalization.py
│       │   └── validation.py
│       │
│       └── worker/
│           ├── __init__.py
│           ├── batch.py
│           ├── clickhouse.py
│           ├── consumer.py
│           ├── dlq.py
│           ├── main.py
│           ├── metrics.py
│           ├── observability.py
│           ├── processor.py
│           └── stats.py
│
├── tests/
│   ├── integration/
│   │   └── test_ingestion_api.py
│   │
│   └── unit/
│       ├── test_batch.py
│       ├── test_dlq.py
│       ├── test_normalization.py
│       ├── test_project_setup.py
│       ├── test_validation.py
│       ├── test_worker_processor.py
│       └── test_worker_stats.py.py
│
├── .env.example
├── .gitignore
├── docker-compose.yml
├── pyproject.toml
└── README.md
```

## Configuration

Configuration is provided through environment variables documented in `.env.example`.

This includes:

* Redpanda version and topic settings
* Kafka broker and consumer configuration
* ClickHouse connection settings
* Worker batch size and flush interval

## Running Pipeline

```bash
docker compose up -d --build
```

Verify services:

```bash
docker compose ps -a
```

Expected:

```text
Redpanda       healthy
redpanda-init  exited successfully
ClickHouse     healthy
```

### Available endpoints:

#### API:

```text
GET  http://localhost:8000/health
GET  http://localhost:8000/metrics
GET  http://localhost:8000/stats
POST http://localhost:8000/api/v1/cdr/raw
```

#### Worker:

```text
GET  http://localhost:8001/health
GET  http://localhost:8001/metrics
GET  http://localhost:8001/internal/stats
```

A valid CDR currently returns:

```json
{
  "status": "accepted",
  "raw_id": "raw-test-001"
}
```

Invalid payloads return structured `400` responses.

## Testing

Once the full stack starts cleanly, a real CDR reaches ClickHouse, and:

```bash
pytest -v
```

## Delivery Semantics

The worker uses at-least-once delivery.

Kafka offsets are committed only after a ClickHouse batch is successfully written or, after exhausted retries, the complete failed batch is acknowledged by the DLQ.

Repeated records use deterministic UUIDv5 identifiers and `ReplacingMergeTree` to support replay-safe storage.