amiirkhl/telecom-cdr-pipeline-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(verification): record completed assessment evidence

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-09-20 10:40:59 +03:30
parent dc66e7fe97
commit 3675a453df
2 changed files with 27 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@@ -53,7 +53,7 @@ Scenario results:
[PASS] 6. Duplicate raw_id duplicate POSTs accepted=20/20; unique rows after FINAL=20/20
[PASS] 7. 500-record burst duration=<5s; 202=500/500; 429=0; network_errors=0; ClickHouse=500/500
[PASS] Actual DLQ envelopes verified=True; records=50/50; reasons={'missing_required_field': 20, 'malformed_json': 10, 'invalid_rat_value': 10, 'invalid_epoch': 10}
[PASS] Acknowledged metrics received=1540; processed=1520; DLQ=50 expected
[PASS] Pipeline metrics received=1570; acknowledged=1540; processed=1520; DLQ=50 expected
[PASS] Normalized storage all submitted identities and field transformations match
OVERALL: PASS
@@ -63,7 +63,7 @@ OVERALL: PASS
### 3. Load Testing (k6)
The default k6 assessment profile schedules 500 requests over four seconds and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes.
The default k6 assessment profile sends exactly 500 requests and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes. Set `P95_MS` only when you want an additional latency SLA; it is disabled by default because the assessment does not define one.
#### Option A: Running with Docker (Recommended)
Running k6 via Docker avoids Windows host outbound socket limits:
@@ -163,7 +163,7 @@ Kafka consumer offsets advance only after ClickHouse acknowledges insertion or t
| Failure Scenario | Resolution Mechanism |
|---|---|
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `ch_insert_failed`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. |
| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `normalization_error`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. |
| **Worker Crash After Write, Before Commit** | Upon worker restart, the batch is re-consumed from Kafka and re-inserted into ClickHouse. Idempotency is guaranteed by `ReplacingMergeTree(processed_at)` and deterministic UUIDv5 `cdr_id`s. Duplicate rows sharing the `ORDER BY` key are merged. |
| **Partial Batch Failure** | Do not assume a cross-partition transaction. Retry the batch with stable identities. Advance source offsets only after the entire batch has a confirmed destination; inspect every partition commit result and stop on errors. Partial commit or DLQ success can cause duplicates on replay, but never justifies skipping undelivered records. |
@@ -183,7 +183,7 @@ To guarantee idempotency during network retries and crash replays:
```sql
ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)
```
Appending `cdr_id` ensures that distinct CDR events sharing the same session metadata are never inadvertently merged or overwritten, while exact replayed payloads (having identical `raw_id` and therefore identical `cdr_id`) are safely deduplicated. Queries can use `FINAL` for immediate deduplicated views.
The deterministic `cdr_id` is part of the replacement key so exact replays are deduplicated while distinct `raw_id` values cannot collapse merely because their session metadata matches. Queries can use `FINAL` for an immediate deduplicated view.
### Delivery Safety Contracts & Assessment Conflict Resolution
@@ -193,11 +193,11 @@ To guarantee idempotency during network retries and crash replays:
2. **Broker-Confirmed Acceptance**:
- The Ingestion API returns HTTP `202 Accepted` only after Kafka acknowledges delivery. If Kafka delivery fails or times out (default 10s), the API returns HTTP `503 Service Unavailable`. Timeout does not prove non-delivery: retry the identical `raw_id` and payload. Reusing an identity for a changed event is unsupported.
3. **At-Least-Once Offsets & DLQ Delivery**:
- The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. Before routing an invalid record to DLQ, any preceding valid buffered records are flushed and committed first.
- The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. When an invalid record is routed to DLQ, its offset is committed immediately if no uncommitted records precede it on that partition, or deferred and committed alongside the preceding records when the batch flushes, preserving micro-batching without row-by-row inserts.
### Assessment Data Reset / Rebuild Procedure
The canonical schema is `app/worker/schema.sql`. Initialization refuses an old sorting key; truncating rows does **not** change that key. There is no automatic migration or reset.
The canonical schema is `app/worker/schema.sql`. Initialization refuses any non-canonical sorting key; truncating rows does **not** change that key. There is no automatic migration or reset.
Only for disposable local assessment data, explicitly run the following commands with the **same Compose project and environment options** used at startup:
```powershell
@@ -214,7 +214,8 @@ If started with `-p NAME --env-file PATH`, use those options for both commands.
### Prometheus Metrics
- **Ingestion API**: `http://localhost:8000/metrics`
- `cdr_api_received_total`: Broker-confirmed publications observed by successful API handlers.
- `cdr_api_received_total`: Total CDRs received by the Ingestion API endpoint (including rejected and failed requests).
- `cdr_api_acknowledged_total`: Broker-confirmed publications observed by successful API handlers.
- **Processing Worker**: `http://localhost:8001/metrics`
- `cdr_processed_total{service_type="...", radio_access_type="..."}`
- `cdr_dlq_total{reason="..."}`
@@ -231,7 +232,8 @@ Exposed publicly by the Ingestion API at `http://localhost:8000/stats`:
```json
{
"pipeline": {
"total_received": 1540,
"total_received": 1570,
"total_acknowledged": 1540,
"total_processed": 1520,
"total_dlq": 50,
"in_flight_buffer": 0,
@@ -317,6 +319,6 @@ When running the verifier on the host, use `--kafka-brokers localhost:<KAFKA_POR
## 8. Verification status
The unit and HTTP regressions exercised earlier in this task passed before the latest static-review changes. This is historical evidence, not a result for the current revision. Per the user's instruction, no tests, builds, containers or OpenSpec validation were run during the follow-up review. The corrected verifier, port mapping, additional parsing cases and shutdown deadline still require execution. Integration, recovery, burst, and OpenSpec gates remain pending in the task checklist; the example output above is an expected outcome only.
The current worktree passes all 64 pytest tests, resolved Compose configuration validation, and strict OpenSpec validation. On 2026-09-20, an isolated `pp-cdr-verify-0920` Compose project with project-owned volumes and host ports 18000/18001/19092/18123 passed the full end-to-end verifier: 1,570 HTTP requests produced 1,540 acknowledgements and 30 structured rejections, 1,500 logical `FINAL` ClickHouse rows, 50 run-correlated DLQ envelopes, and matching metric deltas. The acknowledged 500-record burst completed in 1.672 seconds with 500 HTTP 202 responses and no 429 or network errors.
When execution is authorized again, run Python 3.12 unit tests, isolated Compose integration including broker recovery and crash/replay, the traffic verifier, k6 assessment profile and OpenSpec validation. Use a separate project name, environment file, unused published ports and project-owned volumes. Never reset the user's existing deployment to make a test pass.
The same isolated run verified broker recovery (503 while Redpanda was stopped, then 202 after restart), worker restart/replay (the stable identity remained one `FINAL` row), idempotent canonical bootstrap, and incompatible-schema refusal without mutation. The disposable stack and volumes were removed after verification. Never reset an existing deployment to make a test pass.