# Telecom Raw CDR Ingestion & Analytics Pipeline

A high-throughput, fault-tolerant streaming data pipeline designed to ingest, buffer, normalize, and store raw telecom Call Detail Records (CDRs) using **pure Python** (framework-less `asyncio`), **Kafka (Redpanda)**, and **ClickHouse**.

Built for **at-least-once delivery**, **backpressure protection**, **deterministic deduplication**, and **operational observability**.

---

## 1. Quickstart

### Prerequisites
- Docker & Docker Compose v2+
- Python 3.12+ (for running the local verification script)

### 1. Start the Pipeline
```bash
docker compose up --build
```
This single command brings up:
- **Redpanda** (Kafka-compatible message broker on port `9092`)
- **Topic Provisioner** (`topic-init`: creates `cdr.raw` and `cdr.raw.dlq` with exact partitions)
- **ClickHouse** (Analytics database on port `8123` HTTP / `9000` Native with auto-initialized schema)
- **Ingestion API** (Port `8000`)
- **Processing Worker** (Port `8001`)

All dependent services declare health checks with `depends_on: { condition: service_healthy }` to eliminate startup race conditions.

### 2. Verify Pipeline Behaviors
In a separate terminal, run the automated traffic generator:
```bash
python scripts/generate_traffic.py
```

Expected verification summary:
```
=== Pipeline Verification Summary ===
Records sent:               1570
Records in ClickHouse:      1000    ✅
Records in DLQ:             50      ✅
API 400 errors:             10      ✅
Duplicate raw_ids resolved: 20      ✅
=====================================
```

### 3. Load Testing (k6)

A k6 load test script is available at [`scripts/load_test.js`](scripts/load_test.js) to simulate high-concurrency ingestion (ramping up to 50 virtual users pushing sustained load).

#### Option A: Running with Docker (Recommended)
Running k6 via Docker avoids Windows host outbound socket limits:

- **Windows (PowerShell / Docker Desktop)**:
  ```powershell
  docker run --rm -v "${PWD}/scripts:/scripts" grafana/k6 run -e API_URL="http://host.docker.internal:8000/api/v1/cdr/raw" /scripts/load_test.js
  ```

- **Linux / macOS (Host Network)**:
  ```bash
  docker run --rm -i --network=host grafana/k6 run - <scripts/load_test.js
  ```

#### Option B: Running with Local k6
If k6 is installed on your machine:

- **Windows (PowerShell)**:
  ```powershell
  k6 run scripts/load_test.js
  # Or using explicit path:
  & "C:\Program Files\k6\k6.exe" run scripts/load_test.js
  ```

- **Linux / macOS**:
  ```bash
  k6 run scripts/load_test.js
  ```

> [!TIP]
> **Windows Ephemeral Port Note**: Under rapid high-concurrency load (>20,000 requests in <60 seconds), Windows may exhaust dynamic outbound ports held in `TIME_WAIT`, throwing `connectex: Only one usage of each socket address is normally permitted`. Running k6 inside Docker (Option A) avoids this host-level limitation.


---

## 2. Architecture & Data Flow

```mermaid
flowchart LR
    NE([Network Element]) -->|POST /api/v1/cdr/raw| API[Ingestion API :8000]
    API -->|Bounded Async Queue| BP[Backpressure Gate]
    BP -->|Key: raw_id| Kafka[(Redpanda: cdr.raw)]

    Kafka -->|Consume| Worker[Processing Worker :8001]

    subgraph WorkerLogic [Worker Pipeline]
        direction TB
        V[Validate & Clean] --> N[Normalize Schema]
        N --> B[Micro-Batch Buffer]
        B --> OC[Commit Offsets after ClickHouse ACK]
    end
    Worker --- WorkerLogic

    WorkerLogic -->|Batch Insert| CH[(ClickHouse: cdr_records)]
    WorkerLogic -.->|Malformed / Errors| DLQ[(Redpanda: cdr.raw.dlq)]
    WorkerLogic --> PromW[Worker /metrics]

    API --> PromA[API /metrics]
    API -->|Query Internal Stats| Worker
    API --> Stats[Public GET /stats]
```

### Component Roles
- **Ingestion API**: A framework-less HTTP/1.1 server built with standard library `asyncio.start_server`. Enforces bounded in-memory async queue backpressure (`429 Too Many Requests`), validates JSON syntax, and publishes to Kafka using `raw_id` as the partition key.
- **Message Broker (Redpanda/Kafka)**:
  - `cdr.raw`: 3 partitions, LZ4 compression, 24-hour retention.
  - `cdr.raw.dlq`: 1 partition, 7-day retention.
- **Processing Worker**: Consumes `cdr.raw`, normalizes subscriber telephone numbers to E.164 (`phonenumbers`), maps radio access technology (RAT), derives deterministic UUIDv5 `cdr_id`s, micro-batches into ClickHouse, and manages Kafka offset commits.
- **ClickHouse**: Columnar storage using `ReplacingMergeTree(processed_at)` partitioned by month and ordered by subscriber metadata and timestamp.

---

## 3. Micro-Batching Strategy

Inserting row-by-row into ClickHouse causes MergeTree merge storms and performance degradation. The processing worker implements micro-batching:

### Flush Conditions (Whichever Occurs First)
1. **Count Threshold**: Accumulated in-memory buffer reaches `BATCH_SIZE` (default: **1,000 records**).
2. **Time Threshold**: Elapsed time since last flush reaches `BATCH_FLUSH_INTERVAL_SECONDS` (default: **2 seconds**).

### Graceful Shutdown (`SIGTERM` / `SIGINT`)
On container stop or deployment:
1. Worker stops polling new messages from Kafka.
2. Any remaining records buffered in memory are flushed to ClickHouse.
3. Kafka offsets for the flushed batch are committed.
4. Worker exits cleanly with exit code 0.
**Zero records buffered in memory are lost on shutdown.**

---

## 4. Offset Management & At-Least-Once Delivery

Kafka consumer offsets are committed **only after** ClickHouse acknowledges the batch insert.

### Failure Scenario Handling

| Failure Scenario | Resolution Mechanism |
|---|---|
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq`, and offsets are committed to prevent pipeline deadlock. |
| **Worker Crash After Write, Before Commit** | Upon worker restart, the batch is re-consumed from Kafka and re-inserted into ClickHouse. Idempotency is guaranteed by `ReplacingMergeTree(processed_at)` and deterministic UUIDv5 `cdr_id`s. Duplicate rows sharing the `ORDER BY` key are merged. |
| **Partial Batch Failure** | All-or-nothing transactional semantics. ClickHouse batch inserts are atomic per block; consumer offsets are never partially committed. |

---

## 5. Deduplication Strategy

To guarantee idempotency during network retries and crash replays:
1. **Deterministic `cdr_id` Generation**:
   Derived using **UUIDv5** with the DNS namespace and `raw_id`:
   ```python
   import uuid
   cdr_id = uuid.uuid5(uuid.NAMESPACE_DNS, raw_id)
   ```
   Identical `raw_id`s always generate the exact same `cdr_id`.
2. **`ReplacingMergeTree(processed_at)` Engine**:
   ```sql
   ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn)
   ```
   ClickHouse collapses duplicate rows sharing the compound order key, keeping only the most recent `processed_at` timestamp. Queries can also use `FINAL` for immediate deduplicated views.

---

## 6. Observability & Monitoring

### Prometheus Metrics
- **Ingestion API**: `http://localhost:8000/metrics`
  - `cdr_api_received_total`: Total CDRs received at the API gate.
- **Processing Worker**: `http://localhost:8001/metrics`
  - `cdr_processed_total{service_type="...", radio_access_type="..."}`
  - `cdr_dlq_total{reason="..."}`
  - `cdr_kafka_to_clickhouse_latency_seconds` (Histogram)
  - `cdr_batch_flush_duration_seconds` (Histogram)
  - `cdr_batch_size_at_flush` (Histogram)
  - `cdr_in_flight_buffer_size` (Gauge)
  - `cdr_kafka_consumer_lag{partition="..."}` (Gauge)

### Unified `/stats` Endpoint
Exposed publicly by the Ingestion API at `http://localhost:8000/stats`:
```json
{
  "pipeline": {
    "total_received": 1570,
    "total_processed": 1500,
    "total_dlq": 50,
    "in_flight_buffer": 0,
    "last_flush_at": "2026-09-19T10:00:00Z"
  },
  "breakdown": {
    "by_service_type": { "4G_DATA": 1000, "5G_DATA": 500 },
    "by_radio_access_type": { "LTE_4G": 1000, "5G_NR": 500 },
    "dlq_by_reason": {
      "missing_required_field": 20,
      "invalid_rat_value": 10,
      "invalid_epoch": 10,
      "normalization_error": 10
    }
  }
}
```

### Analytical Queries (ClickHouse)
Connect via HTTP `http://localhost:8123` or CLI:
```sql
-- 1. Total count by service type
SELECT service_type, count() AS total
FROM telecom.cdr_records
GROUP BY service_type;

-- 2. Average session duration by Radio Access Type
SELECT radio_access_type, avg(duration_seconds) AS avg_duration
FROM telecom.cdr_records
GROUP BY radio_access_type;

-- 3. Top cells by total traffic volume
SELECT cell_id, sum(total_bytes) AS total_traffic
FROM telecom.cdr_records
GROUP BY cell_id
ORDER BY total_traffic DESC
LIMIT 10;
```

---

## 7. Configuration Reference

All application behavior is driven by environment variables:

| Variable | Default | Component | Description |
|---|---|---|---|
| `API_HOST` | `0.0.0.0` | API | HTTP host interface to bind |
| `API_PORT` | `8000` | API | Ingestion API port |
| `API_QUEUE_MAX_SIZE` | `10000` | API | Bounded queue capacity (returns 429 when full) |
| `WORKER_STATS_URL` | `http://worker:8001/internal/stats` | API | Internal worker stats endpoint for aggregation |
| `WORKER_INSTANCE` | `worker-1` | Worker | Worker instance identifier for DLQ tracing |
| `WORKER_STATS_HOST` | `0.0.0.0` | Worker | Worker internal HTTP host |
| `WORKER_STATS_PORT` | `8001` | Worker | Worker internal HTTP port |
| `KAFKA_BOOTSTRAP_SERVERS`| `localhost:9092` | All | Kafka / Redpanda broker address |
| `KAFKA_RAW_TOPIC` | `cdr.raw` | All | Raw CDR topic name (3 partitions) |
| `KAFKA_DLQ_TOPIC` | `cdr.raw.dlq` | Worker | Dead Letter Queue topic name |
| `KAFKA_CONSUMER_GROUP` | `cdr-processing-group` | Worker | Kafka consumer group ID |
| `CLICKHOUSE_HOST` | `localhost` | Worker | ClickHouse host address |
| `CLICKHOUSE_PORT` | `8123` | Worker | ClickHouse HTTP port |
| `CLICKHOUSE_DATABASE` | `telecom` | Worker | Target database name |
| `CLICKHOUSE_TABLE` | `cdr_records` | Worker | Target table name |
| `BATCH_SIZE` | `1000` | Worker | Number of records before triggering a micro-batch flush |
| `BATCH_FLUSH_INTERVAL_SECONDS` | `2.0` | Worker | Maximum seconds before triggering a micro-batch flush |
| `CH_RETRY_MAX_ATTEMPTS` | `3` | Worker | Maximum ClickHouse insert retry attempts |
| `CH_RETRY_INITIAL_DELAY_MS` | `500` | Worker | Initial retry backoff delay in milliseconds |
| `CH_RETRY_BACKOFF_FACTOR` | `2.0` | Worker | Exponential retry multiplier |
| `LOG_LEVEL` | `INFO` | All | Logging level (`DEBUG`, `INFO`, `WARN`, `ERROR`) |