From 3675a453dfc814de1c2494f7faf34de227308793 Mon Sep 17 00:00:00 2001 From: Amirhossein Khalili Date: Sun, 20 Sep 2026 10:40:59 +0330 Subject: [PATCH] docs(verification): record completed assessment evidence --- README.md | 22 +++++++------- .../changes/telecom-cdr-pipeline/tasks.md | 30 +++++++++---------- 2 files changed, 27 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 481240c..f679aa4 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ Scenario results: [PASS] 6. Duplicate raw_id duplicate POSTs accepted=20/20; unique rows after FINAL=20/20 [PASS] 7. 500-record burst duration=<5s; 202=500/500; 429=0; network_errors=0; ClickHouse=500/500 [PASS] Actual DLQ envelopes verified=True; records=50/50; reasons={'missing_required_field': 20, 'malformed_json': 10, 'invalid_rat_value': 10, 'invalid_epoch': 10} - [PASS] Acknowledged metrics received=1540; processed=1520; DLQ=50 expected + [PASS] Pipeline metrics received=1570; acknowledged=1540; processed=1520; DLQ=50 expected [PASS] Normalized storage all submitted identities and field transformations match OVERALL: PASS @@ -63,7 +63,7 @@ OVERALL: PASS ### 3. Load Testing (k6) -The default k6 assessment profile schedules 500 requests over four seconds and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes. +The default k6 assessment profile sends exactly 500 requests and requires all acknowledgements within five seconds. Set `PROFILE=stress` for the separate sustained-load profile. k6 checks API acceptance; the Python verifier checks durable ClickHouse and DLQ outcomes. Set `P95_MS` only when you want an additional latency SLA; it is disabled by default because the assessment does not define one. #### Option A: Running with Docker (Recommended) Running k6 via Docker avoids Windows host outbound socket limits: @@ -163,7 +163,7 @@ Kafka consumer offsets advance only after ClickHouse acknowledges insertion or t | Failure Scenario | Resolution Mechanism | |---|---| -| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `ch_insert_failed`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. | +| **ClickHouse Write Timeout** | Worker retries the insert with exponential backoff (`CH_RETRY_MAX_ATTEMPTS=3`, initial delay `500ms`, factor `2.0`). If all retries fail, the entire batch is wrapped in DLQ envelopes, published to `cdr.raw.dlq` with reason `normalization_error`, and offsets advance only after all DLQ acknowledgements. A failed DLQ delivery stops the worker without committing that batch. | | **Worker Crash After Write, Before Commit** | Upon worker restart, the batch is re-consumed from Kafka and re-inserted into ClickHouse. Idempotency is guaranteed by `ReplacingMergeTree(processed_at)` and deterministic UUIDv5 `cdr_id`s. Duplicate rows sharing the `ORDER BY` key are merged. | | **Partial Batch Failure** | Do not assume a cross-partition transaction. Retry the batch with stable identities. Advance source offsets only after the entire batch has a confirmed destination; inspect every partition commit result and stop on errors. Partial commit or DLQ success can cause duplicates on replay, but never justifies skipping undelivered records. | @@ -183,7 +183,7 @@ To guarantee idempotency during network retries and crash replays: ```sql ORDER BY (service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id) ``` - Appending `cdr_id` ensures that distinct CDR events sharing the same session metadata are never inadvertently merged or overwritten, while exact replayed payloads (having identical `raw_id` and therefore identical `cdr_id`) are safely deduplicated. Queries can use `FINAL` for immediate deduplicated views. + The deterministic `cdr_id` is part of the replacement key so exact replays are deduplicated while distinct `raw_id` values cannot collapse merely because their session metadata matches. Queries can use `FINAL` for an immediate deduplicated view. ### Delivery Safety Contracts & Assessment Conflict Resolution @@ -193,11 +193,11 @@ To guarantee idempotency during network retries and crash replays: 2. **Broker-Confirmed Acceptance**: - The Ingestion API returns HTTP `202 Accepted` only after Kafka acknowledges delivery. If Kafka delivery fails or times out (default 10s), the API returns HTTP `503 Service Unavailable`. Timeout does not prove non-delivery: retry the identical `raw_id` and payload. Reusing an identity for a changed event is unsupported. 3. **At-Least-Once Offsets & DLQ Delivery**: - - The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. Before routing an invalid record to DLQ, any preceding valid buffered records are flushed and committed first. + - The worker commits Kafka offsets per-partition only after ClickHouse acknowledges batch insertion, or after DLQ acknowledges message persistence. When an invalid record is routed to DLQ, its offset is committed immediately if no uncommitted records precede it on that partition, or deferred and committed alongside the preceding records when the batch flushes, preserving micro-batching without row-by-row inserts. ### Assessment Data Reset / Rebuild Procedure -The canonical schema is `app/worker/schema.sql`. Initialization refuses an old sorting key; truncating rows does **not** change that key. There is no automatic migration or reset. +The canonical schema is `app/worker/schema.sql`. Initialization refuses any non-canonical sorting key; truncating rows does **not** change that key. There is no automatic migration or reset. Only for disposable local assessment data, explicitly run the following commands with the **same Compose project and environment options** used at startup: ```powershell @@ -214,7 +214,8 @@ If started with `-p NAME --env-file PATH`, use those options for both commands. ### Prometheus Metrics - **Ingestion API**: `http://localhost:8000/metrics` - - `cdr_api_received_total`: Broker-confirmed publications observed by successful API handlers. + - `cdr_api_received_total`: Total CDRs received by the Ingestion API endpoint (including rejected and failed requests). + - `cdr_api_acknowledged_total`: Broker-confirmed publications observed by successful API handlers. - **Processing Worker**: `http://localhost:8001/metrics` - `cdr_processed_total{service_type="...", radio_access_type="..."}` - `cdr_dlq_total{reason="..."}` @@ -231,7 +232,8 @@ Exposed publicly by the Ingestion API at `http://localhost:8000/stats`: ```json { "pipeline": { - "total_received": 1540, + "total_received": 1570, + "total_acknowledged": 1540, "total_processed": 1520, "total_dlq": 50, "in_flight_buffer": 0, @@ -317,6 +319,6 @@ When running the verifier on the host, use `--kafka-brokers localhost: