docs(pipeline): align delivery contracts and verification requirements
This commit is contained in:
131
README.md
131
README.md
@@ -10,20 +10,22 @@ Built for **at-least-once delivery**, **backpressure protection**, **determinist
|
||||
|
||||
### Prerequisites
|
||||
- Docker & Docker Compose v2+
|
||||
- Python 3.12+ (for running the local verification script)
|
||||
- Python 3.12+ with `python -m pip install -r requirements.txt` (for local tests and the traffic verifier)
|
||||
|
||||
### 1. Start the Pipeline
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
No `.env` file is required. To override defaults, copy `.env.example` to `.env` and edit it.
|
||||
|
||||
This single command brings up:
|
||||
- **Redpanda** (Kafka-compatible message broker on port `9092`)
|
||||
- **Topic Provisioner** (`topic-init`: creates `cdr.raw` and `cdr.raw.dlq` with exact partitions)
|
||||
- **Infrastructure Initializer** (`init`: creates/verifies topics and the canonical `app/worker/schema.sql` table definition)
|
||||
- **ClickHouse** (Analytics database on port `8123` HTTP / `9000` Native with auto-initialized schema)
|
||||
- **Ingestion API** (Port `8000`)
|
||||
- **Processing Worker** (Port `8001`)
|
||||
|
||||
All dependent services declare health checks with `depends_on: { condition: service_healthy }` to eliminate startup race conditions.
|
||||
The initializer waits for broker/database health; API and worker wait for `init` to complete successfully. Existing incompatible schemas or topic settings fail initialization without deleting data.
|
||||
|
||||
### 2. Verify Pipeline Behaviors
|
||||
In a separate terminal, run the automated traffic generator:
|
||||
@@ -31,20 +33,37 @@ In a separate terminal, run the automated traffic generator:
|
||||
python scripts/generate_traffic.py
|
||||
```
|
||||
|
||||
Expected verification summary:
|
||||
Illustrative expected verification summary (not evidence that the current revision passed):
|
||||
```
|
||||
=== Pipeline Verification Summary ===
|
||||
Records sent: 1570
|
||||
Records in ClickHouse: 1000 ✅
|
||||
Records in DLQ: 50 ✅
|
||||
API 400 errors: 10 ✅
|
||||
Duplicate raw_ids resolved: 20 ✅
|
||||
=====================================
|
||||
=== Assessment Verification Summary ===
|
||||
Records/requests sent: 1570 (expected 1,570)
|
||||
All HTTP statuses: 202:1540, 400:30
|
||||
Run-scoped ClickHouse rows: 1500 (expected 1,500 unique valid+burst rows)
|
||||
/stats received delta: 1540
|
||||
/stats processed delta: 1520
|
||||
/stats DLQ delta: 50 (expected 50)
|
||||
/stats in-flight buffer: 0
|
||||
|
||||
Scenario results:
|
||||
[PASS] 1. Valid records HTTP 202=1000/1000; ClickHouse=1000/1000
|
||||
[PASS] 2. Missing caller_num HTTP=400:20; DLQ missing_required_field=20/20
|
||||
[PASS] 3. Malformed JSON HTTP 400=10/10; structured JSON errors=10/10
|
||||
[PASS] 4. Invalid RAT HTTP 202=10/10; DLQ invalid_rat_value=10/10
|
||||
[PASS] 5. Invalid epoch HTTP 202=10/10; DLQ invalid_epoch=10/10
|
||||
[PASS] 6. Duplicate raw_id duplicate POSTs accepted=20/20; unique rows after FINAL=20/20
|
||||
[PASS] 7. 500-record burst duration=<5s; 202=500/500; 429=0; network_errors=0; ClickHouse=500/500
|
||||
[PASS] Actual DLQ envelopes verified=True; records=50/50; reasons={'missing_required_field': 20, 'malformed_json': 10, 'invalid_rat_value': 10, 'invalid_epoch': 10}
|
||||
[PASS] Acknowledged metrics received=1540; processed=1520; DLQ=50 expected
|
||||
[PASS] Normalized storage all submitted identities and field transformations match
|
||||
|
||||
OVERALL: PASS
|
||||
============================================================
|
||||
```
|
||||
|
||||
|
||||
### 3. Load Testing (k6)
|
||||
|
||||
A k6 load test script is available at [`scripts/load_test.js`](scripts/load_test.js) to simulate high-concurrency ingestion (ramping up to 50 virtual users pushing sustained load).
|
||||
The default k6 assessment profile schedules 500 requests over four seconds and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes.
|
||||
|
||||
#### Option A: Running with Docker (Recommended)
|
||||
Running k6 via Docker avoids Windows host outbound socket limits:
|
||||
@@ -85,7 +104,7 @@ If k6 is installed on your machine:
|
||||
```mermaid
|
||||
flowchart LR
|
||||
NE([Network Element]) -->|POST /api/v1/cdr/raw| API[Ingestion API :8000]
|
||||
API -->|Bounded Async Queue| BP[Backpressure Gate]
|
||||
API -->|Bounded Pending Deliveries| BP[Backpressure Gate]
|
||||
BP -->|Key: raw_id| Kafka[(Redpanda: cdr.raw)]
|
||||
|
||||
Kafka -->|Consume| Worker[Processing Worker :8001]
|
||||
@@ -108,7 +127,7 @@ flowchart LR
|
||||
```
|
||||
|
||||
### Component Roles
|
||||
- **Ingestion API**: A framework-less HTTP/1.1 server built with standard library `asyncio.start_server`. Enforces bounded in-memory async queue backpressure (`429 Too Many Requests`), validates JSON syntax, and publishes to Kafka using `raw_id` as the partition key.
|
||||
- **Ingestion API**: A framework-less HTTP/1.1 server built with standard library `asyncio.start_server`. Bounds outstanding Kafka publications for backpressure (`429 Too Many Requests`), validates JSON syntax and field types, and publishes to Kafka using `raw_id` as the partition key.
|
||||
- **Message Broker (Redpanda/Kafka)**:
|
||||
- `cdr.raw`: 3 partitions, LZ4 compression, 24-hour retention.
|
||||
- `cdr.raw.dlq`: 1 partition, 7-day retention.
|
||||
@@ -130,22 +149,23 @@ On container stop or deployment:
|
||||
1. Worker stops polling new messages from Kafka.
|
||||
2. Any remaining records buffered in memory are flushed to ClickHouse.
|
||||
3. Kafka offsets for the flushed batch are committed.
|
||||
4. Worker exits cleanly with exit code 0.
|
||||
**Zero records buffered in memory are lost on shutdown.**
|
||||
4. Worker exits with code 0 only when storage and commits succeed. A destination/commit failure exits nonzero and retains uncommitted source offsets for replay.
|
||||
|
||||
The Linux containers enforce a 45-second shutdown budget and a 60-second Compose grace period. On forced termination, uncommitted records replay on restart. The API stops admission and drains broker acknowledgements before exit. The worker deadline uses POSIX signals; Windows-host worker execution does not enforce that alarm.
|
||||
|
||||
---
|
||||
|
||||
## 4. Offset Management & At-Least-Once Delivery
|
||||
|
||||
Kafka consumer offsets are committed **only after** ClickHouse acknowledges the batch insert.
|
||||
Kafka consumer offsets advance only after ClickHouse acknowledges insertion or the DLQ acknowledges delivery. Both automatic offset storage and automatic commits are disabled. Commits specify the next processed offset for each affected partition.
|
||||
|
||||
### Failure Scenario Handling
|
||||
|
||||
| Failure Scenario | Resolution Mechanism |
|
||||
|---|---|
|
||||
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq`, and offsets are committed to prevent pipeline deadlock. |
|
||||
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `ch_insert_failed`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. |
|
||||
| **Worker Crash After Write, Before Commit** | Upon worker restart, the batch is re-consumed from Kafka and re-inserted into ClickHouse. Idempotency is guaranteed by `ReplacingMergeTree(processed_at)` and deterministic UUIDv5 `cdr_id`s. Duplicate rows sharing the `ORDER BY` key are merged. |
|
||||
| **Partial Batch Failure** | All-or-nothing transactional semantics. ClickHouse batch inserts are atomic per block; consumer offsets are never partially committed. |
|
||||
| **Partial Batch Failure** | Do not assume a cross-partition transaction. Retry the batch with stable identities. Advance source offsets only after the entire batch has a confirmed destination; inspect every partition commit result and stop on errors. Partial commit or DLQ success can cause duplicates on replay, but never justifies skipping undelivered records. |
|
||||
|
||||
---
|
||||
|
||||
@@ -161,9 +181,32 @@ To guarantee idempotency during network retries and crash replays:
|
||||
Identical `raw_id`s always generate the exact same `cdr_id`.
|
||||
2. **`ReplacingMergeTree(processed_at)` Engine**:
|
||||
```sql
|
||||
ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn)
|
||||
ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)
|
||||
```
|
||||
ClickHouse collapses duplicate rows sharing the compound order key, keeping only the most recent `processed_at` timestamp. Queries can also use `FINAL` for immediate deduplicated views.
|
||||
Appending `cdr_id` ensures that distinct CDR events sharing the same session metadata are never inadvertently merged or overwritten, while exact replayed payloads (having identical `raw_id` and therefore identical `cdr_id`) are safely deduplicated. Queries can use `FINAL` for immediate deduplicated views.
|
||||
|
||||
### Delivery Safety Contracts & Assessment Conflict Resolution
|
||||
|
||||
1. **API Validation Boundary**:
|
||||
- The assessment specification contained conflicting requirements: Scenario 2 expected 20 records with missing `caller_num` in the DLQ, while the API specification required strict input validation rejecting missing required fields or invalid types with HTTP `400 Bad Request`.
|
||||
- **Resolution**: The Ingestion API enforces strict wire validation (rejecting missing required fields, non-string/non-integer types, and malformed JSON with structured `400` errors). To verify the worker's DLQ routing for malformed and missing-field records without compromising the API gate, `generate_traffic.py` directly injects 20 missing-field records and 10 malformed JSON records into Kafka, producing 50 DLQ records for the resolved scenarios. These are in addition to 1,570 HTTP requests; the 500-record burst contributes 500 additional unique stored rows. Missing Kafka evidence makes verification fail, never fall back to counters.
|
||||
2. **Broker-Confirmed Acceptance**:
|
||||
- The Ingestion API returns HTTP `202 Accepted` only after Kafka acknowledges delivery. If Kafka delivery fails or times out (default 10s), the API returns HTTP `503 Service Unavailable`. Timeout does not prove non-delivery: retry the identical `raw_id` and payload. Reusing an identity for a changed event is unsupported.
|
||||
3. **At-Least-Once Offsets & DLQ Delivery**:
|
||||
- The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. Before routing an invalid record to DLQ, any preceding valid buffered records are flushed and committed first.
|
||||
|
||||
### Assessment Data Reset / Rebuild Procedure
|
||||
|
||||
The canonical schema is `app/worker/schema.sql`. Initialization refuses an old sorting key; truncating rows does **not** change that key. There is no automatic migration or reset.
|
||||
|
||||
Only for disposable local assessment data, explicitly run the following commands with the **same Compose project and environment options** used at startup:
|
||||
```powershell
|
||||
# Destructive: deletes this Compose project's Kafka and ClickHouse volumes.
|
||||
docker compose down --volumes
|
||||
docker compose up --build -d
|
||||
```
|
||||
If started with `-p NAME --env-file PATH`, use those options for both commands. Do not use this procedure for data that must be preserved. No reset has been performed by the static-review follow-up.
|
||||
|
||||
|
||||
---
|
||||
|
||||
@@ -171,7 +214,7 @@ To guarantee idempotency during network retries and crash replays:
|
||||
|
||||
### Prometheus Metrics
|
||||
- **Ingestion API**: `http://localhost:8000/metrics`
|
||||
- `cdr_api_received_total`: Total CDRs received at the API gate.
|
||||
- `cdr_api_received_total`: Broker-confirmed publications observed by successful API handlers.
|
||||
- **Processing Worker**: `http://localhost:8001/metrics`
|
||||
- `cdr_processed_total{service_type="...", radio_access_type="..."}`
|
||||
- `cdr_dlq_total{reason="..."}`
|
||||
@@ -181,25 +224,27 @@ To guarantee idempotency during network retries and crash replays:
|
||||
- `cdr_in_flight_buffer_size` (Gauge)
|
||||
- `cdr_kafka_consumer_lag{partition="..."}` (Gauge)
|
||||
|
||||
Counters measure physical acknowledged processing/delivery operations, including replay. `FINAL` queries measure logical records. Kafka latency uses the message timestamp; lag is high watermark minus committed next offset and refreshes every five seconds. A worker restart resets its in-memory counters, so run the verifier against a quiet stack without restarts.
|
||||
|
||||
### Unified `/stats` Endpoint
|
||||
Exposed publicly by the Ingestion API at `http://localhost:8000/stats`:
|
||||
```json
|
||||
{
|
||||
"pipeline": {
|
||||
"total_received": 1570,
|
||||
"total_processed": 1500,
|
||||
"total_received": 1540,
|
||||
"total_processed": 1520,
|
||||
"total_dlq": 50,
|
||||
"in_flight_buffer": 0,
|
||||
"last_flush_at": "2026-09-19T10:00:00Z"
|
||||
},
|
||||
"breakdown": {
|
||||
"by_service_type": { "4G_DATA": 1000, "5G_DATA": 500 },
|
||||
"by_radio_access_type": { "LTE_4G": 1000, "5G_NR": 500 },
|
||||
"by_service_type": { "4G_DATA": 1020, "5G_DATA": 500 },
|
||||
"by_radio_access_type": { "LTE_4G": 1020, "5G_NR": 500 },
|
||||
"dlq_by_reason": {
|
||||
"missing_required_field": 20,
|
||||
"invalid_rat_value": 10,
|
||||
"invalid_epoch": 10,
|
||||
"normalization_error": 10
|
||||
"malformed_json": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -210,17 +255,17 @@ Connect via HTTP `http://localhost:8123` or CLI:
|
||||
```sql
|
||||
-- 1. Total count by service type
|
||||
SELECT service_type, count() AS total
|
||||
FROM telecom.cdr_records
|
||||
FROM telecom.cdr_records FINAL
|
||||
GROUP BY service_type;
|
||||
|
||||
-- 2. Average session duration by Radio Access Type
|
||||
SELECT radio_access_type, avg(duration_seconds) AS avg_duration
|
||||
FROM telecom.cdr_records
|
||||
FROM telecom.cdr_records FINAL
|
||||
GROUP BY radio_access_type;
|
||||
|
||||
-- 3. Top cells by total traffic volume
|
||||
SELECT cell_id, sum(total_bytes) AS total_traffic
|
||||
FROM telecom.cdr_records
|
||||
FROM telecom.cdr_records FINAL
|
||||
GROUP BY cell_id
|
||||
ORDER BY total_traffic DESC
|
||||
LIMIT 10;
|
||||
@@ -230,13 +275,15 @@ LIMIT 10;
|
||||
|
||||
## 7. Configuration Reference
|
||||
|
||||
All application behavior is driven by environment variables:
|
||||
Compose reads optional `.env` overrides. Python scripts read exported environment variables, not `.env` automatically. Within Compose, ClickHouse always listens at `clickhouse:8123`; `CLICKHOUSE_PORT` controls only the published host port. For standalone Python clients it specifies their destination port. With a custom external ClickHouse, use explicit Compose service overrides for its internal address/port.
|
||||
|
||||
When running the verifier on the host, use `--kafka-brokers localhost:<KAFKA_PORT>` and the matching `--api-url` / `--clickhouse-url`. Run it without unrelated producers so metric deltas remain meaningful.
|
||||
|
||||
| Variable | Default | Component | Description |
|
||||
|---|---|---|---|
|
||||
| `API_HOST` | `0.0.0.0` | API | HTTP host interface to bind |
|
||||
| `API_PORT` | `8000` | API | Ingestion API port |
|
||||
| `API_QUEUE_MAX_SIZE` | `10000` | API | Bounded queue capacity (returns 429 when full) |
|
||||
| `API_QUEUE_MAX_SIZE` | `10000` | API | Outstanding publications, held until delivery completes |
|
||||
| `WORKER_STATS_URL` | `http://worker:8001/internal/stats` | API | Internal worker stats endpoint for aggregation |
|
||||
| `WORKER_INSTANCE` | `worker-1` | Worker | Worker instance identifier for DLQ tracing |
|
||||
| `WORKER_STATS_HOST` | `0.0.0.0` | Worker | Worker internal HTTP host |
|
||||
@@ -246,7 +293,7 @@ All application behavior is driven by environment variables:
|
||||
| `KAFKA_DLQ_TOPIC` | `cdr.raw.dlq` | Worker | Dead Letter Queue topic name |
|
||||
| `KAFKA_CONSUMER_GROUP` | `cdr-processing-group` | Worker | Kafka consumer group ID |
|
||||
| `CLICKHOUSE_HOST` | `localhost` | Worker | ClickHouse host address |
|
||||
| `CLICKHOUSE_PORT` | `8123` | Worker | ClickHouse HTTP port |
|
||||
| `CLICKHOUSE_PORT` | `8123` | Compose / standalone client | Published host HTTP port / standalone destination port |
|
||||
| `CLICKHOUSE_DATABASE` | `telecom` | Worker | Target database name |
|
||||
| `CLICKHOUSE_TABLE` | `cdr_records` | Worker | Target table name |
|
||||
| `BATCH_SIZE` | `1000` | Worker | Number of records before triggering a micro-batch flush |
|
||||
@@ -255,3 +302,21 @@ All application behavior is driven by environment variables:
|
||||
| `CH_RETRY_INITIAL_DELAY_MS` | `500` | Worker | Initial retry backoff delay in milliseconds |
|
||||
| `CH_RETRY_BACKOFF_FACTOR` | `2.0` | Worker | Exponential retry multiplier |
|
||||
| `LOG_LEVEL` | `INFO` | All | Logging level (`DEBUG`, `INFO`, `WARN`, `ERROR`) |
|
||||
|
||||
| `API_MAX_BODY_BYTES` | `65536` | API | Maximum request body; oversized requests return 413 |
|
||||
| `API_READ_TIMEOUT_SECONDS` | `10` | API | Request read deadline; timeout returns 408 |
|
||||
| `KAFKA_DELIVERY_TIMEOUT_SECONDS` | `10` | API / worker | Broker acknowledgement deadline |
|
||||
| `SHUTDOWN_TIMEOUT_SECONDS` | `45` | API / Linux worker | Graceful shutdown budget |
|
||||
| `KAFKA_AUTO_OFFSET_RESET` | `earliest` | Worker | Initial position when no committed offset exists |
|
||||
| `KAFKA_PORT` | `9092` | Compose | External broker listener and published port |
|
||||
| `KAFKA_ADVERTISED_HOST` | `localhost` | Compose | Broker address advertised to external clients |
|
||||
| `REDPANDA_ADMIN_PORT` | `9644` | Compose | Published Redpanda administration port |
|
||||
| `CLICKHOUSE_NATIVE_PORT` | `9000` | Compose | Published native ClickHouse port |
|
||||
| `CLICKHOUSE_USER` | `default` | Database / clients | Database user |
|
||||
| `CLICKHOUSE_PASSWORD` | empty | Database / clients | Database password |
|
||||
|
||||
## 8. Verification status
|
||||
|
||||
The unit and HTTP regressions exercised earlier in this task passed before the latest static-review changes. This is historical evidence, not a result for the current revision. Per the user's instruction, no tests, builds, containers or OpenSpec validation were run during the follow-up review. The corrected verifier, port mapping, additional parsing cases and shutdown deadline still require execution. Integration, recovery, burst, and OpenSpec gates remain pending in the task checklist; the example output above is an expected outcome only.
|
||||
|
||||
When execution is authorized again, run Python 3.12 unit tests, isolated Compose integration including broker recovery and crash/replay, the traffic verifier, k6 assessment profile and OpenSpec validation. Use a separate project name, environment file, unused published ports and project-owned volumes. Never reset the user's existing deployment to make a test pass.
|
||||
|
||||
@@ -26,7 +26,7 @@ The architecture is constrained to pure Python without web frameworks (FastAPI,
|
||||
|
||||
### 1. Framework-less `asyncio` HTTP Server
|
||||
- **Choice**: Implement the HTTP listener using `asyncio.start_server` with a lightweight, zero-dependency HTTP/1.1 request parser.
|
||||
- **Rationale**: Strictly satisfies the constraint of "pure Python without frameworks" while delivering high concurrent connection handling and native integration with `asyncio.Queue` for backpressure.
|
||||
- **Rationale**: Strictly satisfies the constraint of "pure Python without frameworks" while delivering high concurrent connection handling and a bounded set of asynchronous delivery futures for backpressure.
|
||||
- **Alternatives Considered**:
|
||||
- `http.server.ThreadingHTTPServer`: Standard library, but thread-per-request model has higher memory overhead and makes bounded async backpressure coordination harder.
|
||||
- `FastAPI`/`Starlette`: Disallowed by project constraints.
|
||||
@@ -40,7 +40,7 @@ The architecture is constrained to pure Python without web frameworks (FastAPI,
|
||||
|
||||
### 3. Kafka Client and Partitioning Strategy
|
||||
- **Choice**: Use `confluent-kafka` with `raw_id` as the message key when publishing to `cdr.raw`.
|
||||
- **Rationale**: `confluent-kafka` is built on `librdkafka` and offers superior throughput over pure Python clients. Using `raw_id` ensures deterministic partition affinity for duplicate CDR submissions while spreading traffic evenly across the 3 partitions via Murmur2 hashing.
|
||||
- **Rationale**: `confluent-kafka` is built on `librdkafka` and offers superior throughput over pure Python clients. Using `raw_id` ensures deterministic partition affinity for duplicate CDR submissions while spreading traffic evenly across the 3 partitions using the configured Kafka client partitioner.
|
||||
- **Alternatives Considered**:
|
||||
- `caller_num` as key: Risk of partition hotspots if certain subscribers generate disproportionate traffic.
|
||||
- `null` (round-robin): Loses record-level partition affinity.
|
||||
@@ -56,7 +56,7 @@ The architecture is constrained to pure Python without web frameworks (FastAPI,
|
||||
- **Choice**: Worker accumulates records into an in-memory batch. Flushes when `len(batch) >= BATCH_SIZE` (default 1,000) or elapsed time >= `BATCH_FLUSH_INTERVAL_SECONDS` (default 2s).
|
||||
- **Failure Handling**:
|
||||
- If ClickHouse write times out, worker retries with exponential backoff up to `CH_RETRY_MAX_ATTEMPTS` (default 3, initial delay 500ms, factor 2.0).
|
||||
- If all retries fail, the entire batch is wrapped and routed to `cdr.raw.dlq`, and offsets are committed to prevent pipeline stall.
|
||||
- If all retries fail, the entire batch is wrapped and routed to `cdr.raw.dlq`, and offsets are committed only after all DLQ acknowledgements.
|
||||
- On `SIGTERM`, consumer stops, remaining buffer is flushed to ClickHouse, offsets are committed, and worker exits cleanly.
|
||||
|
||||
### 6. Structured JSON Logging
|
||||
@@ -65,7 +65,20 @@ The architecture is constrained to pure Python without web frameworks (FastAPI,
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk] High ingestion bursts saturate in-memory queue** → **[Mitigation]** The bounded `asyncio.Queue` rejects requests with `429 Too Many Requests` as soon as capacity is reached, protecting process memory.
|
||||
- **[Risk] ClickHouse outage stalls worker** → **[Mitigation]** Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed to prevent infinite consumer lag loops.
|
||||
- **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`.
|
||||
- **[Risk] High ingestion bursts saturate in-memory queue** → **[Mitigation]** The bounded outstanding-publication set rejects requests with `429 Too Many Requests` as soon as capacity is reached, protecting process memory.
|
||||
- **[Risk] ClickHouse outage stalls worker** → **[Mitigation]** Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed only after delivery acknowledgements; a delivery/commit failure stops consumption.
|
||||
- **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`.
|
||||
- **[Risk] Phone number parsing performance overhead** → **[Mitigation]** Use `phonenumbers` library with fast-path prefix checks (`00`, `011`, `+`) before full parsing.
|
||||
|
||||
## Corrected delivery contracts
|
||||
- API validates UTF-8, finite JSON, required non-null fields and exact types before admission. Semantic RAT, range, phone and IPv4 checks remain in the worker. Return structured 400 errors; 413 for bodies exceeding API_MAX_BODY_BYTES (65536), 408 for reads exceeding API_READ_TIMEOUT_SECONDS (10).
|
||||
- Producer returns concurrent delivery futures and polls callbacks continuously. Queued plus outstanding publications are bounded by API_QUEUE_MAX_SIZE. Return 202 after acknowledgement, 429 at capacity, 503 on failure/timeout (KAFKA_DELIVERY_TIMEOUT_SECONDS=10). Timed-out delivery is ambiguous; retry identical raw_id and payload. Recover without a sticky health gate.
|
||||
- Worker disables both automatic offset storage and commits. Before routing an invalid record, flush older buffered records. Await DLQ acknowledgements, then commit explicit next offsets per partition. Failed delivery/commit stops processing nonzero without advancing unsafe offsets; never classify commit failures as ClickHouse failures.
|
||||
- ClickHouse retries at most three times with exponential delays between attempts. Exhausted batches retain original payloads in ch_insert_failed envelopes. Partial DLQ success permits duplicate replay, never missing source records.
|
||||
- raw_id identifies immutable events; replay payloads must be identical. Include cdr_id in the replacement key, use UTC DateTimes and native IPv4 values, retain monthly partitioning and 90-day TTL. Refuse an incompatible existing schema; documented explicit reset only.
|
||||
- Stop API admission and drain acknowledgements on SIGTERM/SIGINT. API and Linux worker shutdown budgets are 45 seconds; Compose grace is 60 seconds. Windows-host workers do not provide the POSIX alarm deadline. Worker fatal cleanup does not flush or commit unsafe state.
|
||||
- Consumer lag is high watermark minus committed next offset (or low watermark before the first commit), refreshed periodically. Latency measures Kafka timestamp to acknowledged ClickHouse write. Counters reflect physical operations, FINAL rows reflect logical events. Escape labels and keep log message plus event.
|
||||
- Compose defaults work without .env, while .env.example uses redpanda:29092. Ports, topics, credentials and schema names propagate to provisioning, clients and health checks. Provisioning verifies existing settings and fails on incompatible infrastructure.
|
||||
- Verification sends 1570 HTTP requests (1540 accepted, 30 rejected), plus 30 direct Kafka injections; expect 1500 logical ClickHouse rows and 50 actual DLQ records. DLQ reads use run identifiers and a separate consumer group. The baseline is single-worker; live migration and conflicting identity updates are excluded.
|
||||
|
||||
Compose keeps the ClickHouse internal HTTP port at 8123; CLICKHOUSE_PORT selects the published host port. The verifier must fail when actual Kafka evidence is unavailable. Current follow-up edits have not been executed or validated by user instruction.
|
||||
|
||||
@@ -30,3 +30,6 @@ Telecom network elements generate high-volume, heterogeneous Call Detail Records
|
||||
- **Dependencies**: `confluent-kafka`, `clickhouse-connect`, `phonenumbers` (no web frameworks).
|
||||
- **Infrastructure**: Kafka/Redpanda broker and ClickHouse database managed via Docker Compose.
|
||||
- **Data Flow**: Network Element → HTTP API → Kafka (`cdr.raw`) → Worker → ClickHouse (`cdr_records`) / DLQ (`cdr.raw.dlq`).
|
||||
|
||||
## Delivery safety correction
|
||||
Broker acknowledgement defines API acceptance. Explicit partition offsets advance only after acknowledged ClickHouse or DLQ storage. The assessment SQL omits identity from its replacement key; append cdr_id to prevent distinct events collapsing. Required-field HTTP rejection and direct Kafka fault injection resolve contradictory assessment examples. Existing assessment data is never reset automatically.
|
||||
|
||||
@@ -9,7 +9,7 @@ The storage layer SHALL provide a database named `telecom` and a table named `cd
|
||||
|
||||
#### Scenario: Database and table initialized idempotently
|
||||
- **WHEN** the initialization script or container entrypoint executes
|
||||
- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
|
||||
- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
|
||||
|
||||
### Requirement: Deduplication on At-Least-Once Replay
|
||||
The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.
|
||||
@@ -24,3 +24,14 @@ The storage layer and worker client SHALL store `source_ip` using the ClickHouse
|
||||
#### Scenario: IPv4 string inserted into table
|
||||
- **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
|
||||
- **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types
|
||||
|
||||
### Requirement: Identity preservation and schema compatibility
|
||||
Initialization SHALL use one canonical schema, verify existing table compatibility and never reset data automatically.
|
||||
|
||||
#### Scenario: Distinct identities share analytical metadata
|
||||
- **WHEN** records have different raw_id values but identical other sorting fields
|
||||
- **THEN** both survive FINAL and background replacement, while exact replay yields one logical row
|
||||
|
||||
#### Scenario: Existing incompatible table
|
||||
- **WHEN** an existing table lacks the corrected identity sorting key
|
||||
- **THEN** initialization fails with explicit assessment reset instructions without deleting data
|
||||
|
||||
@@ -9,11 +9,11 @@ The Ingestion API SHALL provide an HTTP endpoint at `POST /api/v1/cdr/raw` to in
|
||||
|
||||
#### Scenario: Valid CDR payload accepted
|
||||
- **WHEN** a client sends a `POST /api/v1/cdr/raw` request with valid JSON containing all required fields (`raw_id`, `device_id`, `caller_num`, `svc`, `rat`, `epoch_start`, `epoch_end`, `up_bytes`, `down_bytes`, `lac`, `ci`) and the internal buffer has capacity
|
||||
- **THEN** the API returns HTTP status `202 Accepted` with a structured JSON confirmation and queues the payload for Kafka publishing
|
||||
- **THEN** the API returns HTTP status `202 Accepted` with a structured JSON confirmation only after Kafka acknowledges delivery; pending publications count against the bounded capacity
|
||||
|
||||
#### Scenario: Malformed JSON payload rejected
|
||||
- **WHEN** a client sends a `POST /api/v1/cdr/raw` request with an unparseable or corrupted JSON body
|
||||
- **THEN** the API returns HTTP status `400 Bad Request` with structured JSON `{"error": "malformed_json", "detail": "<error_message>"}`
|
||||
- **THEN** the API returns HTTP status `400 Bad Request` with structured JSON `{"error": "invalid_json", "detail": "<error_message>"}`
|
||||
|
||||
#### Scenario: Missing required fields rejected
|
||||
- **WHEN** a client sends a `POST /api/v1/cdr/raw` request missing one or more required fields (such as `caller_num` or `raw_id`)
|
||||
@@ -53,3 +53,14 @@ The Ingestion API SHALL expose a human-readable JSON status endpoint at `GET /st
|
||||
#### Scenario: Unified stats requested
|
||||
- **WHEN** an operator issues a `GET /stats` request
|
||||
- **THEN** the API queries the Worker's internal stats endpoint, aggregates `total_received` with worker metrics (`total_processed`, `total_dlq`, `in_flight_buffer`, `last_flush_at`, and breakdown maps), and returns HTTP status `200 OK` with structured JSON
|
||||
|
||||
### Requirement: Acknowledged bounded acceptance and shutdown
|
||||
The API SHALL hold admission capacity until Kafka delivery completes, continuously service delivery callbacks, and drain on shutdown.
|
||||
|
||||
#### Scenario: Failure, timeout or saturation
|
||||
- **WHEN** publication fails or exceeds the configured 10-second default deadline, or outstanding capacity is full
|
||||
- **THEN** return structured 503 or 429 respectively; never acknowledge an unconfirmed publication, and allow subsequent requests after broker recovery
|
||||
|
||||
#### Scenario: Invalid types and bounded requests
|
||||
- **WHEN** a request contains invalid UTF-8, non-finite JSON, wrong field types, an oversized body, or exceeds the read deadline
|
||||
- **THEN** return structured 400, 413, or 408 respectively without publishing; default body/read limits are 65536 bytes and 10 seconds
|
||||
|
||||
@@ -27,4 +27,4 @@ All messages published to `cdr.raw.dlq` SHALL be wrapped in a standardized JSON
|
||||
|
||||
#### Scenario: Failure reason taxonomy enforced
|
||||
- **WHEN** any record is routed to `cdr.raw.dlq`
|
||||
- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, or `normalization_error`
|
||||
- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`, or `ch_insert_failed` (a documented assessment extension for exhausted storage retries)
|
||||
|
||||
@@ -30,3 +30,14 @@ All pipeline services SHALL emit log lines formatted as structured JSON to stdou
|
||||
#### Scenario: Operational event logged
|
||||
- **WHEN** any service logs an event (info, warning, or error)
|
||||
- **THEN** the log output is a single-line JSON object containing at minimum `timestamp` (ISO 8601 UTC), `level` (INFO, WARN, ERROR), `service` (`ingestion-api` or `processing-worker`), `raw_id` (record identifier or null), and `event` (short descriptive string)
|
||||
|
||||
### Requirement: Measured and truthful telemetry
|
||||
Metrics SHALL count acknowledged destination operations, measure Kafka timestamp-to-write latency and periodically update partition lag from committed offsets and high watermarks.
|
||||
|
||||
#### Scenario: Delivery and scrape
|
||||
- **WHEN** records complete storage and metrics are scraped
|
||||
- **THEN** latency observations and partition gauges are populated, failed DLQ attempts do not increment counters, and label values are correctly escaped
|
||||
|
||||
#### Scenario: Failure diagnostics
|
||||
- **WHEN** a structured event is logged
|
||||
- **THEN** preserve the human-readable message including failure details alongside the event identifier
|
||||
|
||||
@@ -95,3 +95,25 @@ The Processing Worker SHALL handle `SIGTERM` and `SIGINT` signals to flush buffe
|
||||
#### Scenario: SIGTERM received during operation
|
||||
- **WHEN** a `SIGTERM` signal is received by the worker process
|
||||
- **THEN** the worker stops consuming new messages, flushes any remaining in-memory buffered records to ClickHouse, commits the associated Kafka offsets, and exits cleanly with exit code 0
|
||||
|
||||
### Requirement: Safe partition progress and confirmed DLQ
|
||||
The worker SHALL disable automatic offset storage, commit explicit next offsets, and await all destination acknowledgements.
|
||||
|
||||
#### Scenario: Valid record precedes an invalid record
|
||||
- **WHEN** an invalid record follows buffered valid records
|
||||
- **THEN** flush and commit those valid records before acknowledging and committing the invalid record to DLQ
|
||||
|
||||
#### Scenario: Destination or commit failure
|
||||
- **WHEN** DLQ delivery or any partition commit fails
|
||||
- **THEN** stop nonzero without advancing unsafe offsets or reclassifying the error as a ClickHouse failure; fatal cleanup does not flush the buffer
|
||||
|
||||
#### Scenario: Exhausted ClickHouse retries
|
||||
- **WHEN** three configured insert attempts fail
|
||||
- **THEN** publish original vendor payloads with ch_insert_failed and commit only after all DLQ deliveries succeed
|
||||
|
||||
### Requirement: Storage-safe normalization
|
||||
The worker SHALL reject numeric booleans, fractional numeric fields, out-of-range UInt64 and UInt32 values including total bytes, invalid IPv4, and timestamps outside DateTime range without crashing.
|
||||
|
||||
#### Scenario: Invalid scalar or conversion
|
||||
- **WHEN** any field fails its type, range or conversion check
|
||||
- **THEN** route only that record to DLQ with a field-specific reason and preserve other valid records
|
||||
|
||||
@@ -13,7 +13,7 @@ The test suite SHALL provide a script that generates synthetic CDR traffic to ex
|
||||
|
||||
#### Scenario: Missing required field validation
|
||||
- **WHEN** the generator sends 20 records missing `caller_num`
|
||||
- **THEN** all 20 records are rejected by the API with HTTP 400 or routed to DLQ with `missing_required_field`
|
||||
- **THEN** all 20 records are rejected by the API with HTTP 400; 20 additional direct Kafka injections verify DLQ `missing_required_field`
|
||||
|
||||
#### Scenario: Malformed JSON validation
|
||||
- **WHEN** the generator sends 10 invalid JSON payloads
|
||||
@@ -46,3 +46,10 @@ The test script SHALL inspect ClickHouse, the DLQ, and API metrics after test ex
|
||||
- Records in DLQ
|
||||
- API 400 errors
|
||||
- Duplicate `raw_id`s resolved
|
||||
|
||||
### Requirement: Independent durable evidence
|
||||
Verification SHALL inspect actual run-correlated DLQ envelopes and FINAL ClickHouse rows instead of inferring delivery from counters.
|
||||
|
||||
#### Scenario: Full assessment run
|
||||
- **WHEN** 1570 HTTP requests and 30 direct Kafka fault injections finish
|
||||
- **THEN** verify 1540 HTTP acknowledgements, 30 structured HTTP rejections, 1500 unique stored rows, 50 actual DLQ records, correct transformations and metrics, and a 500-record burst under five seconds; unavailable evidence is failure
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
## 1. Project Setup & Common Utilities
|
||||
|
||||
- [x] 1.1 Create project structure, `requirements.txt` (with `confluent-kafka`, `clickhouse-connect`, `phonenumbers`), and `.env.example` defining all environment variables and verify file presence.
|
||||
- [ ] 1.1 Create project structure, `requirements.txt` (with `confluent-kafka`, `clickhouse-connect`, `phonenumbers`), and `.env.example` defining all environment variables and verify file presence.
|
||||
- [x] 1.2 Implement standard library structured JSON logging module emitting `timestamp`, `level`, `service`, `raw_id`, and `event`, and verify output formatting with unit test.
|
||||
- [x] 1.3 Create idempotent ClickHouse initialization script (`init.sql`) creating database `telecom` and `ReplacingMergeTree` table `cdr_records` with TTL and indexing settings, and verify schema correctness.
|
||||
- [ ] 1.3 Create idempotent ClickHouse canonical initialization schema (`app/worker/schema.sql`) creating database `telecom` and `ReplacingMergeTree` table `cdr_records` with TTL and indexing settings, and verify schema correctness.
|
||||
|
||||
## 2. Framework-less Async Ingestion API
|
||||
|
||||
- [x] 2.1 Implement lightweight HTTP/1.1 server using standard library `asyncio.start_server` with route dispatching for `POST /api/v1/cdr/raw`, `GET /metrics`, and `GET /stats`.
|
||||
- [x] 2.2 Implement bounded async queue (`asyncio.Queue`) backpressure: verify `429 Too Many Requests` when queue is saturated and `202 Accepted` when enqueued.
|
||||
- [x] 2.2 Implement bounded outstanding-delivery backpressure: verify `429 Too Many Requests` at capacity and `202 Accepted` only after broker acknowledgement.
|
||||
- [x] 2.3 Implement incoming payload validation: verify `400 Bad Request` with structured JSON error `{ "error": "...", "detail": "..." }` on malformed JSON or missing required fields.
|
||||
- [x] 2.4 Integrate `confluent-kafka` producer publishing to `cdr.raw` using `raw_id` as message key, and return `503 Service Unavailable` if Kafka is unreachable.
|
||||
- [x] 2.5 Implement Prometheus metrics endpoint (`cdr_api_received_total`) and public `GET /stats` endpoint that fetches worker runtime stats via internal HTTP and returns unified JSON.
|
||||
@@ -15,20 +15,33 @@
|
||||
## 3. Processing Worker Implementation
|
||||
|
||||
- [x] 3.1 Implement Kafka consumer loop with `confluent-kafka` consuming from `cdr.raw` and DLQ producer publishing to `cdr.raw.dlq`.
|
||||
- [x] 3.2 Implement safe parser routing malformed records to DLQ with structured envelope and reason taxonomy (`missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`).
|
||||
- [x] 3.3 Implement schema normalizer: E.164 phone conversion via `phonenumbers`, RAT mapping (`1`→`3G`, `6`→`LTE_4G`, `9`→`5G_NR`), `cell_id` derivation (`lac_ci`), and deterministic UUIDv5 `cdr_id` generation.
|
||||
- [ ] 3.2 Implement safe parser routing malformed records to DLQ with structured envelope and reason taxonomy (`missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`).
|
||||
- [ ] 3.3 Implement schema normalizer: E.164 phone conversion via `phonenumbers`, RAT mapping (`1`→`3G`, `6`→`LTE_4G`, `9`→`5G_NR`), `cell_id` derivation (`lac_ci`), and deterministic UUIDv5 `cdr_id` generation.
|
||||
- [x] 3.4 Implement micro-batching buffer flushing on 1,000 records or 2 seconds elapsed, batch-inserting into ClickHouse via `clickhouse-connect`, and committing offsets only after ClickHouse ACK.
|
||||
- [x] 3.5 Implement exponential backoff retry for ClickHouse write timeouts (`CH_RETRY_MAX_ATTEMPTS`, `CH_RETRY_INITIAL_DELAY_MS`, `CH_RETRY_BACKOFF_FACTOR`) with fallback to DLQ.
|
||||
- [x] 3.6 Implement worker internal HTTP stats endpoint (`GET /internal/stats`) and worker Prometheus metrics (`cdr_processed_total`, `cdr_dlq_total`, histograms, gauges).
|
||||
- [x] 3.7 Implement `SIGTERM` and `SIGINT` signal handlers to stop consuming, flush remaining in-memory buffer to ClickHouse, commit offsets, and exit cleanly without data loss.
|
||||
- [ ] 3.7 Implement `SIGTERM` and `SIGINT` signal handlers to stop consuming, flush remaining in-memory buffer to ClickHouse, commit offsets, and exit cleanly without data loss.
|
||||
|
||||
## 4. Containerization & Orchestration
|
||||
|
||||
- [x] 4.1 Create lightweight Dockerfiles for Ingestion API and Processing Worker without framework dependencies.
|
||||
- [x] 4.2 Create `docker-compose.yml` defining Kafka/Redpanda, ClickHouse, Ingestion API, and Processing Worker with healthcheck blocks and `depends_on: { condition: service_healthy }`.
|
||||
- [ ] 4.2 Create `docker-compose.yml` defining Kafka/Redpanda, ClickHouse, Ingestion API, and Processing Worker with healthcheck blocks and `depends_on: { condition: service_healthy }`.
|
||||
|
||||
## 5. Verification & Test Traffic Generator
|
||||
|
||||
- [x] 5.1 Implement traffic generator script covering the 7 mandatory scenarios (valid records, missing fields, malformed JSON, bad RAT, invalid epoch, duplicates, burst load).
|
||||
- [x] 5.2 Implement automated verification summary reporting verifying ClickHouse counts, DLQ counts, API 400 errors, and duplicate resolution.
|
||||
- [ ] 5.1 Implement traffic generator script covering the 7 mandatory scenarios (valid records, missing fields, malformed JSON, bad RAT, invalid epoch, duplicates, burst load).
|
||||
- [ ] 5.2 Implement automated verification summary reporting verifying ClickHouse counts, DLQ counts, API 400 errors, and duplicate resolution.
|
||||
- [x] 5.3 Write comprehensive `README.md` covering quickstart, architecture, micro-batching, offset management, deduplication, monitoring, and configuration reference.
|
||||
|
||||
## 6. Delivery Safety Regression Gates
|
||||
|
||||
- [ ] 6.1 Add and pass unit regressions for confirmed API delivery, shutdown, strict validation and HTTP limits.
|
||||
- [ ] 6.2 Add and pass worker regressions for partition ordering, DLQ failure, commit failure, replay, batching, range checks and telemetry.
|
||||
- [ ] 6.3 Run isolated Compose integration with independent ports/storage, fresh defaults, schema collision/replay and failure recovery checks.
|
||||
- [ ] 6.4 Run actual DLQ traffic verification and the acknowledged 500-record burst; record evidence in README.
|
||||
- [ ] 6.5 Validate OpenSpec and reconcile all task completion claims with evidence.
|
||||
|
||||
|
||||
## Current evidence boundary
|
||||
|
||||
Follow-up changes were reviewed and edited without running verification, as requested by the user. Unchecked items require renewed evidence for the current files; this does not imply their implementation is absent. Historical unit passes must not be used to mark changed code or integration/recovery/load scenarios complete. In particular, a verifier run without actual Kafka evidence is not an acceptance result.
|
||||
|
||||
Reference in New Issue
Block a user