144 lines
2.4 KiB
Markdown
144 lines
2.4 KiB
Markdown
# Telecom CDR Pipeline
|
|
|
|
High-throughput telecom Call Detail Record ingestion and analytics pipeline.
|
|
|
|
## Architecture
|
|
|
|
The system consists of:
|
|
|
|
* FastAPI ingestion API
|
|
* Redpanda as a Kafka-compatible message broker
|
|
* Processing and normalization worker
|
|
* ClickHouse analytical storage
|
|
* Dead Letter Queue for failed records
|
|
|
|
Current data flow:
|
|
|
|
```text
|
|
Network Element
|
|
↓
|
|
FastAPI API
|
|
↓
|
|
Validation
|
|
↓
|
|
202 Accepted
|
|
```
|
|
|
|
Infrastructure already available:
|
|
|
|
```text
|
|
Redpanda
|
|
├── cdr.raw
|
|
└── cdr.raw.dlq
|
|
|
|
ClickHouse
|
|
└── telecom.cdr_records
|
|
```
|
|
|
|
Kafka publishing and worker processing are not connected yet.
|
|
|
|
## Current Features
|
|
|
|
* Dockerized Redpanda and ClickHouse
|
|
* Automatic and idempotent Kafka topic provisioning
|
|
* Automatic ClickHouse schema initialization
|
|
* Persistent Redpanda and ClickHouse volumes
|
|
* Raw and normalized CDR domain models
|
|
* Required-field and type validation
|
|
* RAT validation and normalization
|
|
* E.164 phone-number normalization
|
|
* Deterministic UUIDv5 `cdr_id` generation
|
|
* Epoch and traffic-value validation
|
|
* FastAPI `POST /api/v1/cdr/raw`
|
|
* FastAPI `GET /health`
|
|
* Structured JSON error responses
|
|
* Unit and API integration tests
|
|
|
|
## Project Structure
|
|
|
|
```text
|
|
src/core/
|
|
├── api/
|
|
│ ├── main.py
|
|
│ └── routes.py
|
|
│
|
|
└── domain/
|
|
├── errors.py
|
|
├── models.py
|
|
├── normalization.py
|
|
└── validation.py
|
|
|
|
infrastructure/
|
|
└── clickhouse/
|
|
└── init.sql
|
|
|
|
tests/
|
|
├── integration/
|
|
└── unit/
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Configuration is provided through environment variables documented in `.env.example`.
|
|
|
|
This includes:
|
|
|
|
* Redpanda version and topic settings
|
|
* Kafka broker and consumer configuration
|
|
* ClickHouse connection settings
|
|
* Worker batch size and flush interval
|
|
|
|
## Running Infrastructure
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
Verify services:
|
|
|
|
```bash
|
|
docker compose ps -a
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Redpanda healthy
|
|
redpanda-init exited successfully
|
|
ClickHouse healthy
|
|
```
|
|
|
|
## Running the API
|
|
|
|
```bash
|
|
uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000
|
|
```
|
|
|
|
Available endpoints:
|
|
|
|
```text
|
|
GET /health
|
|
POST /api/v1/cdr/raw
|
|
```
|
|
|
|
A valid CDR currently returns:
|
|
|
|
```json
|
|
{
|
|
"status": "accepted",
|
|
"raw_id": "raw-test-001"
|
|
}
|
|
```
|
|
|
|
Invalid payloads return structured `400` responses.
|
|
|
|
## Testing
|
|
|
|
Run:
|
|
|
|
```bash
|
|
pytest -v
|
|
```
|
|
|
|
The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.
|