258 lines
10 KiB
Markdown
258 lines
10 KiB
Markdown
# Telecom Raw CDR Ingestion & Analytics Pipeline
|
|
|
|
A high-throughput, fault-tolerant streaming data pipeline designed to ingest, buffer, normalize, and store raw telecom Call Detail Records (CDRs) using **pure Python** (framework-less `asyncio`), **Kafka (Redpanda)**, and **ClickHouse**.
|
|
|
|
Built for **at-least-once delivery**, **backpressure protection**, **deterministic deduplication**, and **operational observability**.
|
|
|
|
---
|
|
|
|
## 1. Quickstart
|
|
|
|
### Prerequisites
|
|
- Docker & Docker Compose v2+
|
|
- Python 3.12+ (for running the local verification script)
|
|
|
|
### 1. Start the Pipeline
|
|
```bash
|
|
docker compose up --build
|
|
```
|
|
This single command brings up:
|
|
- **Redpanda** (Kafka-compatible message broker on port `9092`)
|
|
- **Topic Provisioner** (`topic-init`: creates `cdr.raw` and `cdr.raw.dlq` with exact partitions)
|
|
- **ClickHouse** (Analytics database on port `8123` HTTP / `9000` Native with auto-initialized schema)
|
|
- **Ingestion API** (Port `8000`)
|
|
- **Processing Worker** (Port `8001`)
|
|
|
|
All dependent services declare health checks with `depends_on: { condition: service_healthy }` to eliminate startup race conditions.
|
|
|
|
### 2. Verify Pipeline Behaviors
|
|
In a separate terminal, run the automated traffic generator:
|
|
```bash
|
|
python scripts/generate_traffic.py
|
|
```
|
|
|
|
Expected verification summary:
|
|
```
|
|
=== Pipeline Verification Summary ===
|
|
Records sent: 1570
|
|
Records in ClickHouse: 1000 ✅
|
|
Records in DLQ: 50 ✅
|
|
API 400 errors: 10 ✅
|
|
Duplicate raw_ids resolved: 20 ✅
|
|
=====================================
|
|
```
|
|
|
|
### 3. Load Testing (k6)
|
|
|
|
A k6 load test script is available at [`scripts/load_test.js`](scripts/load_test.js) to simulate high-concurrency ingestion (ramping up to 50 virtual users pushing sustained load).
|
|
|
|
#### Option A: Running with Docker (Recommended)
|
|
Running k6 via Docker avoids Windows host outbound socket limits:
|
|
|
|
- **Windows (PowerShell / Docker Desktop)**:
|
|
```powershell
|
|
docker run --rm -v "${PWD}/scripts:/scripts" grafana/k6 run -e API_URL="http://host.docker.internal:8000/api/v1/cdr/raw" /scripts/load_test.js
|
|
```
|
|
|
|
- **Linux / macOS (Host Network)**:
|
|
```bash
|
|
docker run --rm -i --network=host grafana/k6 run - <scripts/load_test.js
|
|
```
|
|
|
|
#### Option B: Running with Local k6
|
|
If k6 is installed on your machine:
|
|
|
|
- **Windows (PowerShell)**:
|
|
```powershell
|
|
k6 run scripts/load_test.js
|
|
# Or using explicit path:
|
|
& "C:\Program Files\k6\k6.exe" run scripts/load_test.js
|
|
```
|
|
|
|
- **Linux / macOS**:
|
|
```bash
|
|
k6 run scripts/load_test.js
|
|
```
|
|
|
|
> [!TIP]
|
|
> **Windows Ephemeral Port Note**: Under rapid high-concurrency load (>20,000 requests in <60 seconds), Windows may exhaust dynamic outbound ports held in `TIME_WAIT`, throwing `connectex: Only one usage of each socket address is normally permitted`. Running k6 inside Docker (Option A) avoids this host-level limitation.
|
|
|
|
|
|
---
|
|
|
|
## 2. Architecture & Data Flow
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
NE([Network Element]) -->|POST /api/v1/cdr/raw| API[Ingestion API :8000]
|
|
API -->|Bounded Async Queue| BP[Backpressure Gate]
|
|
BP -->|Key: raw_id| Kafka[(Redpanda: cdr.raw)]
|
|
|
|
Kafka -->|Consume| Worker[Processing Worker :8001]
|
|
|
|
subgraph WorkerLogic [Worker Pipeline]
|
|
direction TB
|
|
V[Validate & Clean] --> N[Normalize Schema]
|
|
N --> B[Micro-Batch Buffer]
|
|
B --> OC[Commit Offsets after ClickHouse ACK]
|
|
end
|
|
Worker --- WorkerLogic
|
|
|
|
WorkerLogic -->|Batch Insert| CH[(ClickHouse: cdr_records)]
|
|
WorkerLogic -.->|Malformed / Errors| DLQ[(Redpanda: cdr.raw.dlq)]
|
|
WorkerLogic --> PromW[Worker /metrics]
|
|
|
|
API --> PromA[API /metrics]
|
|
API -->|Query Internal Stats| Worker
|
|
API --> Stats[Public GET /stats]
|
|
```
|
|
|
|
### Component Roles
|
|
- **Ingestion API**: A framework-less HTTP/1.1 server built with standard library `asyncio.start_server`. Enforces bounded in-memory async queue backpressure (`429 Too Many Requests`), validates JSON syntax, and publishes to Kafka using `raw_id` as the partition key.
|
|
- **Message Broker (Redpanda/Kafka)**:
|
|
- `cdr.raw`: 3 partitions, LZ4 compression, 24-hour retention.
|
|
- `cdr.raw.dlq`: 1 partition, 7-day retention.
|
|
- **Processing Worker**: Consumes `cdr.raw`, normalizes subscriber telephone numbers to E.164 (`phonenumbers`), maps radio access technology (RAT), derives deterministic UUIDv5 `cdr_id`s, micro-batches into ClickHouse, and manages Kafka offset commits.
|
|
- **ClickHouse**: Columnar storage using `ReplacingMergeTree(processed_at)` partitioned by month and ordered by subscriber metadata and timestamp.
|
|
|
|
---
|
|
|
|
## 3. Micro-Batching Strategy
|
|
|
|
Inserting row-by-row into ClickHouse causes MergeTree merge storms and performance degradation. The processing worker implements micro-batching:
|
|
|
|
### Flush Conditions (Whichever Occurs First)
|
|
1. **Count Threshold**: Accumulated in-memory buffer reaches `BATCH_SIZE` (default: **1,000 records**).
|
|
2. **Time Threshold**: Elapsed time since last flush reaches `BATCH_FLUSH_INTERVAL_SECONDS` (default: **2 seconds**).
|
|
|
|
### Graceful Shutdown (`SIGTERM` / `SIGINT`)
|
|
On container stop or deployment:
|
|
1. Worker stops polling new messages from Kafka.
|
|
2. Any remaining records buffered in memory are flushed to ClickHouse.
|
|
3. Kafka offsets for the flushed batch are committed.
|
|
4. Worker exits cleanly with exit code 0.
|
|
**Zero records buffered in memory are lost on shutdown.**
|
|
|
|
---
|
|
|
|
## 4. Offset Management & At-Least-Once Delivery
|
|
|
|
Kafka consumer offsets are committed **only after** ClickHouse acknowledges the batch insert.
|
|
|
|
### Failure Scenario Handling
|
|
|
|
| Failure Scenario | Resolution Mechanism |
|
|
|---|---|
|
|
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq`, and offsets are committed to prevent pipeline deadlock. |
|
|
| **Worker Crash After Write, Before Commit** | Upon worker restart, the batch is re-consumed from Kafka and re-inserted into ClickHouse. Idempotency is guaranteed by `ReplacingMergeTree(processed_at)` and deterministic UUIDv5 `cdr_id`s. Duplicate rows sharing the `ORDER BY` key are merged. |
|
|
| **Partial Batch Failure** | All-or-nothing transactional semantics. ClickHouse batch inserts are atomic per block; consumer offsets are never partially committed. |
|
|
|
|
---
|
|
|
|
## 5. Deduplication Strategy
|
|
|
|
To guarantee idempotency during network retries and crash replays:
|
|
1. **Deterministic `cdr_id` Generation**:
|
|
Derived using **UUIDv5** with the DNS namespace and `raw_id`:
|
|
```python
|
|
import uuid
|
|
cdr_id = uuid.uuid5(uuid.NAMESPACE_DNS, raw_id)
|
|
```
|
|
Identical `raw_id`s always generate the exact same `cdr_id`.
|
|
2. **`ReplacingMergeTree(processed_at)` Engine**:
|
|
```sql
|
|
ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn)
|
|
```
|
|
ClickHouse collapses duplicate rows sharing the compound order key, keeping only the most recent `processed_at` timestamp. Queries can also use `FINAL` for immediate deduplicated views.
|
|
|
|
---
|
|
|
|
## 6. Observability & Monitoring
|
|
|
|
### Prometheus Metrics
|
|
- **Ingestion API**: `http://localhost:8000/metrics`
|
|
- `cdr_api_received_total`: Total CDRs received at the API gate.
|
|
- **Processing Worker**: `http://localhost:8001/metrics`
|
|
- `cdr_processed_total{service_type="...", radio_access_type="..."}`
|
|
- `cdr_dlq_total{reason="..."}`
|
|
- `cdr_kafka_to_clickhouse_latency_seconds` (Histogram)
|
|
- `cdr_batch_flush_duration_seconds` (Histogram)
|
|
- `cdr_batch_size_at_flush` (Histogram)
|
|
- `cdr_in_flight_buffer_size` (Gauge)
|
|
- `cdr_kafka_consumer_lag{partition="..."}` (Gauge)
|
|
|
|
### Unified `/stats` Endpoint
|
|
Exposed publicly by the Ingestion API at `http://localhost:8000/stats`:
|
|
```json
|
|
{
|
|
"pipeline": {
|
|
"total_received": 1570,
|
|
"total_processed": 1500,
|
|
"total_dlq": 50,
|
|
"in_flight_buffer": 0,
|
|
"last_flush_at": "2026-09-19T10:00:00Z"
|
|
},
|
|
"breakdown": {
|
|
"by_service_type": { "4G_DATA": 1000, "5G_DATA": 500 },
|
|
"by_radio_access_type": { "LTE_4G": 1000, "5G_NR": 500 },
|
|
"dlq_by_reason": {
|
|
"missing_required_field": 20,
|
|
"invalid_rat_value": 10,
|
|
"invalid_epoch": 10,
|
|
"normalization_error": 10
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Analytical Queries (ClickHouse)
|
|
Connect via HTTP `http://localhost:8123` or CLI:
|
|
```sql
|
|
-- 1. Total count by service type
|
|
SELECT service_type, count() AS total
|
|
FROM telecom.cdr_records
|
|
GROUP BY service_type;
|
|
|
|
-- 2. Average session duration by Radio Access Type
|
|
SELECT radio_access_type, avg(duration_seconds) AS avg_duration
|
|
FROM telecom.cdr_records
|
|
GROUP BY radio_access_type;
|
|
|
|
-- 3. Top cells by total traffic volume
|
|
SELECT cell_id, sum(total_bytes) AS total_traffic
|
|
FROM telecom.cdr_records
|
|
GROUP BY cell_id
|
|
ORDER BY total_traffic DESC
|
|
LIMIT 10;
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Configuration Reference
|
|
|
|
All application behavior is driven by environment variables:
|
|
|
|
| Variable | Default | Component | Description |
|
|
|---|---|---|---|
|
|
| `API_HOST` | `0.0.0.0` | API | HTTP host interface to bind |
|
|
| `API_PORT` | `8000` | API | Ingestion API port |
|
|
| `API_QUEUE_MAX_SIZE` | `10000` | API | Bounded queue capacity (returns 429 when full) |
|
|
| `WORKER_STATS_URL` | `http://worker:8001/internal/stats` | API | Internal worker stats endpoint for aggregation |
|
|
| `WORKER_INSTANCE` | `worker-1` | Worker | Worker instance identifier for DLQ tracing |
|
|
| `WORKER_STATS_HOST` | `0.0.0.0` | Worker | Worker internal HTTP host |
|
|
| `WORKER_STATS_PORT` | `8001` | Worker | Worker internal HTTP port |
|
|
| `KAFKA_BOOTSTRAP_SERVERS`| `localhost:9092` | All | Kafka / Redpanda broker address |
|
|
| `KAFKA_RAW_TOPIC` | `cdr.raw` | All | Raw CDR topic name (3 partitions) |
|
|
| `KAFKA_DLQ_TOPIC` | `cdr.raw.dlq` | Worker | Dead Letter Queue topic name |
|
|
| `KAFKA_CONSUMER_GROUP` | `cdr-processing-group` | Worker | Kafka consumer group ID |
|
|
| `CLICKHOUSE_HOST` | `localhost` | Worker | ClickHouse host address |
|
|
| `CLICKHOUSE_PORT` | `8123` | Worker | ClickHouse HTTP port |
|
|
| `CLICKHOUSE_DATABASE` | `telecom` | Worker | Target database name |
|
|
| `CLICKHOUSE_TABLE` | `cdr_records` | Worker | Target table name |
|
|
| `BATCH_SIZE` | `1000` | Worker | Number of records before triggering a micro-batch flush |
|
|
| `BATCH_FLUSH_INTERVAL_SECONDS` | `2.0` | Worker | Maximum seconds before triggering a micro-batch flush |
|
|
| `CH_RETRY_MAX_ATTEMPTS` | `3` | Worker | Maximum ClickHouse insert retry attempts |
|
|
| `CH_RETRY_INITIAL_DELAY_MS` | `500` | Worker | Initial retry backoff delay in milliseconds |
|
|
| `CH_RETRY_BACKOFF_FACTOR` | `2.0` | Worker | Exponential retry multiplier |
|
|
| `LOG_LEVEL` | `INFO` | All | Logging level (`DEBUG`, `INFO`, `WARN`, `ERROR`) |
|