# Telecom CDR Pipeline

High-throughput telecom Call Detail Record ingestion and analytics pipeline.

## Architecture

The system consists of:

* FastAPI ingestion API
* Redpanda as a Kafka-compatible message broker
* Processing and normalization worker
* ClickHouse analytical storage
* Dead Letter Queue for failed records

Current data flow:

```text
Network Element
    ↓
FastAPI API
    ↓
Validation
    ↓
Bounded Async Queue
    ↓
Redpanda / cdr.raw
```

Infrastructure already available:

```text
Redpanda
├── cdr.raw
└── cdr.raw.dlq

ClickHouse
└── telecom.cdr_records
```

worker processing is not connected yet.

## Current Features

* Dockerized Redpanda and ClickHouse
* Automatic and idempotent Kafka topic provisioning
* Automatic ClickHouse schema initialization
* Persistent Redpanda and ClickHouse volumes
* Raw and normalized CDR domain models
* Required-field and type validation
* RAT validation and normalization
* E.164 phone-number normalization
* Deterministic UUIDv5 `cdr_id` generation
* Epoch and traffic-value validation
* FastAPI `POST /api/v1/cdr/raw`
* FastAPI `GET /health`
* Structured JSON error responses
* Unit and API integration tests
* Bounded asynchronous ingestion queue for backpressure
* Long-lived asynchronous Kafka producer
* Kafka delivery acknowledgement before `202 Accepted`
* `429 Too Many Requests` on ingestion queue saturation
* `503 Service Unavailable` on Kafka delivery failure

## Project Structure

```text
src/core/
├── config.py
│
├── api/
│   ├── __init__.py
│   ├── kafka_publisher.py
│   ├── main.py
│   ├── publisher.py
│   └── routes.py
│
└── domain/
    ├── __init__.py
    ├── errors.py
    ├── models.py
    ├── normalization.py
    └── validation.py

infrastructure/
└── clickhouse/
    └── init.sql

tests/
├── integration/
└── unit/
```

## Configuration

Configuration is provided through environment variables documented in `.env.example`.

This includes:

* Redpanda version and topic settings
* Kafka broker and consumer configuration
* ClickHouse connection settings
* Worker batch size and flush interval

## Running Infrastructure

```bash
docker compose up -d
```

Verify services:

```bash
docker compose ps -a
```

Expected:

```text
Redpanda       healthy
redpanda-init  exited successfully
ClickHouse     healthy
```

## Running the API

```bash
uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000
```

Available endpoints:

```text
GET  /health
POST /api/v1/cdr/raw
```

A valid CDR currently returns:

```json
{
  "status": "accepted",
  "raw_id": "raw-test-001"
}
```

Invalid payloads return structured `400` responses.

## Testing

Run:

```bash
pytest -v
```

The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.