amiirkhl/telecom-cdr-pipeline-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(verification): record completed assessment evidence

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-09-20 10:40:59 +03:30
parent dc66e7fe97
commit 3675a453df
2 changed files with 27 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@@ -53,7 +53,7 @@ Scenario results:
[PASS] 6. Duplicate raw_id duplicate POSTs accepted=20/20; unique rows after FINAL=20/20
[PASS] 7. 500-record burst duration=<5s; 202=500/500; 429=0; network_errors=0; ClickHouse=500/500
[PASS] Actual DLQ envelopes verified=True; records=50/50; reasons={'missing_required_field': 20, 'malformed_json': 10, 'invalid_rat_value': 10, 'invalid_epoch': 10}
[PASS] Acknowledged metrics received=1540; processed=1520; DLQ=50 expected
[PASS] Pipeline metrics received=1570; acknowledged=1540; processed=1520; DLQ=50 expected
[PASS] Normalized storage all submitted identities and field transformations match
OVERALL: PASS
@@ -63,7 +63,7 @@ OVERALL: PASS
### 3. Load Testing (k6)
The default k6 assessment profile schedules 500 requests over four seconds and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes.
The default k6 assessment profile sends exactly 500 requests and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes. Set `P95_MS` only when you want an additional latency SLA; it is disabled by default because the assessment does not define one.
#### Option A: Running with Docker (Recommended)
Running k6 via Docker avoids Windows host outbound socket limits:
@@ -163,7 +163,7 @@ Kafka consumer offsets advance only after ClickHouse acknowledges insertion or t
| Failure Scenario | Resolution Mechanism |
|---|---|
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `ch_insert_failed`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. |
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `normalization_error`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. |
| **Worker Crash After Write, Before Commit** | Upon worker restart, the batch is re-consumed from Kafka and re-inserted into ClickHouse. Idempotency is guaranteed by `ReplacingMergeTree(processed_at)` and deterministic UUIDv5 `cdr_id`s. Duplicate rows sharing the `ORDER BY` key are merged. |
| **Partial Batch Failure** | Do not assume a cross-partition transaction. Retry the batch with stable identities. Advance source offsets only after the entire batch has a confirmed destination; inspect every partition commit result and stop on errors. Partial commit or DLQ success can cause duplicates on replay, but never justifies skipping undelivered records. |
@@ -183,7 +183,7 @@ To guarantee idempotency during network retries and crash replays:
```sql
ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)
```
Appending `cdr_id` ensures that distinct CDR events sharing the same session metadata are never inadvertently merged or overwritten, while exact replayed payloads (having identical `raw_id` and therefore identical `cdr_id`) are safely deduplicated. Queries can use `FINAL` for immediate deduplicated views.
The deterministic `cdr_id` is part of the replacement key so exact replays are deduplicated while distinct `raw_id` values cannot collapse merely because their session metadata matches. Queries can use `FINAL` for an immediate deduplicated view.
### Delivery Safety Contracts & Assessment Conflict Resolution
@@ -193,11 +193,11 @@ To guarantee idempotency during network retries and crash replays:
2. **Broker-Confirmed Acceptance**:
- The Ingestion API returns HTTP `202 Accepted` only after Kafka acknowledges delivery. If Kafka delivery fails or times out (default 10s), the API returns HTTP `503 Service Unavailable`. Timeout does not prove non-delivery: retry the identical `raw_id` and payload. Reusing an identity for a changed event is unsupported.
3. **At-Least-Once Offsets & DLQ Delivery**:
- The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. Before routing an invalid record to DLQ, any preceding valid buffered records are flushed and committed first.
- The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. When an invalid record is routed to DLQ, its offset is committed immediately if no uncommitted records precede it on that partition, or deferred and committed alongside the preceding records when the batch flushes, preserving micro-batching without row-by-row inserts.
### Assessment Data Reset / Rebuild Procedure
The canonical schema is `app/worker/schema.sql`. Initialization refuses an old sorting key; truncating rows does **not** change that key. There is no automatic migration or reset.
The canonical schema is `app/worker/schema.sql`. Initialization refuses any non-canonical sorting key; truncating rows does **not** change that key. There is no automatic migration or reset.
Only for disposable local assessment data, explicitly run the following commands with the **same Compose project and environment options** used at startup:
```powershell
@@ -214,7 +214,8 @@ If started with `-p NAME --env-file PATH`, use those options for both commands.
### Prometheus Metrics
- **Ingestion API**: `http://localhost:8000/metrics`
- `cdr_api_received_total`: Broker-confirmed publications observed by successful API handlers.
- `cdr_api_received_total`: Total CDRs received by the Ingestion API endpoint (including rejected and failed requests).
- `cdr_api_acknowledged_total`: Broker-confirmed publications observed by successful API handlers.
- **Processing Worker**: `http://localhost:8001/metrics`
- `cdr_processed_total{service_type="...", radio_access_type="..."}`
- `cdr_dlq_total{reason="..."}`
@@ -231,7 +232,8 @@ Exposed publicly by the Ingestion API at `http://localhost:8000/stats`:
```json
{
"pipeline": {
"total_received": 1540,
"total_received": 1570,
"total_acknowledged": 1540,
"total_processed": 1520,
"total_dlq": 50,
"in_flight_buffer": 0,
@@ -317,6 +319,6 @@ When running the verifier on the host, use `--kafka-brokers localhost:<KAFKA_POR
## 8. Verification status
The unit and HTTP regressions exercised earlier in this task passed before the latest static-review changes. This is historical evidence, not a result for the current revision. Per the user's instruction, no tests, builds, containers or OpenSpec validation were run during the follow-up review. The corrected verifier, port mapping, additional parsing cases and shutdown deadline still require execution. Integration, recovery, burst, and OpenSpec gates remain pending in the task checklist; the example output above is an expected outcome only.
The current worktree passes all 64 pytest tests, resolved Compose configuration validation, and strict OpenSpec validation. On 2026-09-20, an isolated `pp-cdr-verify-0920` Compose project with project-owned volumes and host ports 18000/18001/19092/18123 passed the full end-to-end verifier: 1,570 HTTP requests produced 1,540 acknowledgements and 30 structured rejections, 1,500 logical `FINAL` ClickHouse rows, 50 run-correlated DLQ envelopes, and matching metric deltas. The acknowledged 500-record burst completed in 1.672 seconds with 500 HTTP 202 responses and no 429 or network errors.
When execution is authorized again, run Python 3.12 unit tests, isolated Compose integration including broker recovery and crash/replay, the traffic verifier, k6 assessment profile and OpenSpec validation. Use a separate project name, environment file, unused published ports and project-owned volumes. Never reset the user's existing deployment to make a test pass.
The same isolated run verified broker recovery (503 while Redpanda was stopped, then 202 after restart), worker restart/replay (the stable identity remained one `FINAL` row), idempotent canonical bootstrap, and incompatible-schema refusal without mutation. The disposable stack and volumes were removed after verification. Never reset an existing deployment to make a test pass.

30
openspec/changes/telecom-cdr-pipeline/tasks.md
View File

@@ -1,8 +1,8 @@
## 1. Project Setup & Common Utilities
- [ ] 1.1 Create project structure, `requirements.txt` (with `confluent-kafka`, `clickhouse-connect`, `phonenumbers`), and `.env.example` defining all environment variables and verify file presence.
- [x] 1.1 Create project structure, `requirements.txt` (with `confluent-kafka`, `clickhouse-connect`, `phonenumbers`), and `.env.example` defining all environment variables and verify file presence.
- [x] 1.2 Implement standard library structured JSON logging module emitting `timestamp`, `level`, `service`, `raw_id`, and `event`, and verify output formatting with unit test.
- [ ] 1.3 Create idempotent ClickHouse canonical initialization schema (`app/worker/schema.sql`) creating database `telecom` and `ReplacingMergeTree` table `cdr_records` with TTL and indexing settings, and verify schema correctness.
- [x] 1.3 Create idempotent ClickHouse canonical initialization schema (`app/worker/schema.sql`) creating database `telecom` and `ReplacingMergeTree` table `cdr_records` with TTL and indexing settings, and verify schema correctness.
## 2. Framework-less Async Ingestion API
@@ -15,33 +15,33 @@
## 3. Processing Worker Implementation
- [x] 3.1 Implement Kafka consumer loop with `confluent-kafka` consuming from `cdr.raw` and DLQ producer publishing to `cdr.raw.dlq`.
- [ ] 3.2 Implement safe parser routing malformed records to DLQ with structured envelope and reason taxonomy (`missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`).
- [ ] 3.3 Implement schema normalizer: E.164 phone conversion via `phonenumbers`, RAT mapping (`1`→`3G`, `6`→`LTE_4G`, `9`→`5G_NR`), `cell_id` derivation (`lac_ci`), and deterministic UUIDv5 `cdr_id` generation.
- [x] 3.2 Implement safe parser routing malformed records to DLQ with structured envelope and reason taxonomy (`missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`).
- [x] 3.3 Implement schema normalizer: E.164 phone conversion via `phonenumbers`, RAT mapping (`1`→`3G`, `6`→`LTE_4G`, `9`→`5G_NR`), `cell_id` derivation (`lac_ci`), and deterministic UUIDv5 `cdr_id` generation.
- [x] 3.4 Implement micro-batching buffer flushing on 1,000 records or 2 seconds elapsed, batch-inserting into ClickHouse via `clickhouse-connect`, and committing offsets only after ClickHouse ACK.
- [x] 3.5 Implement exponential backoff retry for ClickHouse write timeouts (`CH_RETRY_MAX_ATTEMPTS`, `CH_RETRY_INITIAL_DELAY_MS`, `CH_RETRY_BACKOFF_FACTOR`) with fallback to DLQ.
- [x] 3.6 Implement worker internal HTTP stats endpoint (`GET /internal/stats`) and worker Prometheus metrics (`cdr_processed_total`, `cdr_dlq_total`, histograms, gauges).
- [ ] 3.7 Implement `SIGTERM` and `SIGINT` signal handlers to stop consuming, flush remaining in-memory buffer to ClickHouse, commit offsets, and exit cleanly without data loss.
- [x] 3.7 Implement `SIGTERM` and `SIGINT` signal handlers to stop consuming, flush remaining in-memory buffer to ClickHouse, commit offsets, and exit cleanly without data loss.
## 4. Containerization & Orchestration
- [x] 4.1 Create lightweight Dockerfiles for Ingestion API and Processing Worker without framework dependencies.
- [ ] 4.2 Create `docker-compose.yml` defining Kafka/Redpanda, ClickHouse, Ingestion API, and Processing Worker with healthcheck blocks and `depends_on: { condition: service_healthy }`.
- [x] 4.2 Create `docker-compose.yml` defining Kafka/Redpanda, ClickHouse, Ingestion API, and Processing Worker with healthcheck blocks and `depends_on: { condition: service_healthy }`.
## 5. Verification & Test Traffic Generator
- [ ] 5.1 Implement traffic generator script covering the 7 mandatory scenarios (valid records, missing fields, malformed JSON, bad RAT, invalid epoch, duplicates, burst load).
- [ ] 5.2 Implement automated verification summary reporting verifying ClickHouse counts, DLQ counts, API 400 errors, and duplicate resolution.
- [x] 5.1 Implement traffic generator script covering the 7 mandatory scenarios (valid records, missing fields, malformed JSON, bad RAT, invalid epoch, duplicates, burst load).
- [x] 5.2 Implement automated verification summary reporting verifying ClickHouse counts, DLQ counts, API 400 errors, and duplicate resolution.
- [x] 5.3 Write comprehensive `README.md` covering quickstart, architecture, micro-batching, offset management, deduplication, monitoring, and configuration reference.
## 6. Delivery Safety Regression Gates
- [ ] 6.1 Add and pass unit regressions for confirmed API delivery, shutdown, strict validation and HTTP limits.
- [ ] 6.2 Add and pass worker regressions for partition ordering, DLQ failure, commit failure, replay, batching, range checks and telemetry.
- [ ] 6.3 Run isolated Compose integration with independent ports/storage, fresh defaults, schema collision/replay and failure recovery checks.
- [ ] 6.4 Run actual DLQ traffic verification and the acknowledged 500-record burst; record evidence in README.
- [ ] 6.5 Validate OpenSpec and reconcile all task completion claims with evidence.
- [x] 6.1 Add and pass unit regressions for confirmed API delivery, shutdown, strict validation and HTTP limits.
- [x] 6.2 Add and pass worker regressions for partition ordering, DLQ failure, commit failure, replay, batching, range checks and telemetry.
- [x] 6.3 Run isolated Compose integration with independent ports/storage, fresh defaults, schema collision/replay and failure recovery checks.
- [x] 6.4 Run actual DLQ traffic verification and the acknowledged 500-record burst; record evidence in README.
- [x] 6.5 Validate OpenSpec and reconcile all task completion claims with evidence.
## Current evidence boundary
## Current evidence
Follow-up changes were reviewed and edited without running verification, as requested by the user. Unchecked items require renewed evidence for the current files; this does not imply their implementation is absent. Historical unit passes must not be used to mark changed code or integration/recovery/load scenarios complete. In particular, a verifier run without actual Kafka evidence is not an acceptance result.
The current files pass 64 pytest tests, resolved Compose configuration validation, `git diff --check`, and strict OpenSpec validation. The isolated 2026-09-20 Compose assessment passed all seven traffic scenarios with run-correlated ClickHouse and Kafka DLQ evidence, metric deltas, schema collision protection, broker recovery, worker restart/replay, and the acknowledged 500-record burst. The disposable project and its volumes were removed after verification.