diff --git a/openspec/changes/telecom-cdr-pipeline/design.md b/openspec/changes/telecom-cdr-pipeline/design.md index 12c7825..346d6b0 100644 --- a/openspec/changes/telecom-cdr-pipeline/design.md +++ b/openspec/changes/telecom-cdr-pipeline/design.md @@ -67,15 +67,15 @@ The architecture is constrained to pure Python without web frameworks (FastAPI, - **[Risk] High ingestion bursts saturate in-memory queue** → **[Mitigation]** The bounded outstanding-publication set rejects requests with `429 Too Many Requests` as soon as capacity is reached, protecting process memory. - **[Risk] ClickHouse outage stalls worker** → **[Mitigation]** Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed only after delivery acknowledgements; a delivery/commit failure stops consumption. -- **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`. +- **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the identity-bearing order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`. - **[Risk] Phone number parsing performance overhead** → **[Mitigation]** Use `phonenumbers` library with fast-path prefix checks (`00`, `011`, `+`) before full parsing. ## Corrected delivery contracts - API validates UTF-8, finite JSON, required non-null fields and exact types before admission. Semantic RAT, range, phone and IPv4 checks remain in the worker. Return structured 400 errors; 413 for bodies exceeding API_MAX_BODY_BYTES (65536), 408 for reads exceeding API_READ_TIMEOUT_SECONDS (10). - Producer returns concurrent delivery futures and polls callbacks continuously. Queued plus outstanding publications are bounded by API_QUEUE_MAX_SIZE. Return 202 after acknowledgement, 429 at capacity, 503 on failure/timeout (KAFKA_DELIVERY_TIMEOUT_SECONDS=10). Timed-out delivery is ambiguous; retry identical raw_id and payload. Recover without a sticky health gate. -- Worker disables both automatic offset storage and commits. Before routing an invalid record, flush older buffered records. Await DLQ acknowledgements, then commit explicit next offsets per partition. Failed delivery/commit stops processing nonzero without advancing unsafe offsets; never classify commit failures as ClickHouse failures. -- ClickHouse retries at most three times with exponential delays between attempts. Exhausted batches retain original payloads in ch_insert_failed envelopes. Partial DLQ success permits duplicate replay, never missing source records. -- raw_id identifies immutable events; replay payloads must be identical. Include cdr_id in the replacement key, use UTC DateTimes and native IPv4 values, retain monthly partitioning and 90-day TTL. Refuse an incompatible existing schema; documented explicit reset only. +- Worker disables both automatic offset storage and commits. Route invalid records to DLQ without prematurely flushing buffer; defer committing invalid record offsets if preceded by uncommitted buffer records on that partition, committing when the batch flushes. Await DLQ acknowledgements, then commit explicit next offsets per partition. Failed delivery/commit stops processing nonzero without advancing unsafe offsets; never classify commit failures as ClickHouse failures. +- ClickHouse retries at most three times with exponential delays between attempts. Exhausted batches retain original payloads in normalization_error envelopes. Partial DLQ success permits duplicate replay, never missing source records. +- raw_id identifies immutable events; replay payloads must be identical. Append deterministic `cdr_id` to the assessment replacement key, use UTC DateTimes and native IPv4 values, and retain monthly partitioning and 90-day TTL. This preserves replay deduplication without replacing distinct events that happen to share all session metadata. Refuse an incompatible existing schema; documented explicit reset only. - Stop API admission and drain acknowledgements on SIGTERM/SIGINT. API and Linux worker shutdown budgets are 45 seconds; Compose grace is 60 seconds. Windows-host workers do not provide the POSIX alarm deadline. Worker fatal cleanup does not flush or commit unsafe state. - Consumer lag is high watermark minus committed next offset (or low watermark before the first commit), refreshed periodically. Latency measures Kafka timestamp to acknowledged ClickHouse write. Counters reflect physical operations, FINAL rows reflect logical events. Escape labels and keep log message plus event. - Compose defaults work without .env, while .env.example uses redpanda:29092. Ports, topics, credentials and schema names propagate to provisioning, clients and health checks. Provisioning verifies existing settings and fails on incompatible infrastructure. diff --git a/openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md b/openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md index ac11839..5a2947b 100644 --- a/openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md +++ b/openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md @@ -16,7 +16,7 @@ The storage layer SHALL support idempotent re-insertion of records without creat #### Scenario: Duplicate record inserted after crash replay - **WHEN** a batch containing previously written records is re-inserted due to consumer restart before offset commit -- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates rows sharing the same order key upon merge or when queried with `FINAL` +- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates exact replays sharing the deterministic `cdr_id`-bearing order key upon merge or when queried with `FINAL`, without collapsing distinct `raw_id` values whose session metadata happens to match ### Requirement: Source IP Column Type Compatibility The storage layer and worker client SHALL store `source_ip` using the ClickHouse `IPv4` data type. @@ -29,9 +29,9 @@ The storage layer and worker client SHALL store `source_ip` using the ClickHouse Initialization SHALL use one canonical schema, verify existing table compatibility and never reset data automatically. #### Scenario: Distinct identities share analytical metadata -- **WHEN** records have different raw_id values but identical other sorting fields -- **THEN** both survive FINAL and background replacement, while exact replay yields one logical row +- **WHEN** records have different `raw_id` values but identical other sorting fields +- **THEN** both survive `FINAL` and background replacement, while an exact replay yields one logical row #### Scenario: Existing incompatible table -- **WHEN** an existing table lacks the corrected identity sorting key +- **WHEN** an existing table does not use the canonical identity-bearing sorting key - **THEN** initialization fails with explicit assessment reset instructions without deleting data diff --git a/openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md b/openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md index 6eacbb9..204839e 100644 --- a/openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md +++ b/openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md @@ -27,4 +27,5 @@ All messages published to `cdr.raw.dlq` SHALL be wrapped in a standardized JSON #### Scenario: Failure reason taxonomy enforced - **WHEN** any record is routed to `cdr.raw.dlq` -- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`, or `ch_insert_failed` (a documented assessment extension for exhausted storage retries) +- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, or `normalization_error` +- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error` diff --git a/openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md b/openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md index bdfa529..901b45e 100644 --- a/openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md +++ b/openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md @@ -102,6 +102,7 @@ The worker SHALL disable automatic offset storage, commit explicit next offsets, #### Scenario: Valid record precedes an invalid record - **WHEN** an invalid record follows buffered valid records - **THEN** flush and commit those valid records before acknowledging and committing the invalid record to DLQ +- **THEN** route the invalid record to DLQ and defer committing its offset until buffered valid records are flushed, or commit immediately if no preceding uncommitted records exist on that partition #### Scenario: Destination or commit failure - **WHEN** DLQ delivery or any partition commit fails @@ -109,7 +110,8 @@ The worker SHALL disable automatic offset storage, commit explicit next offsets, #### Scenario: Exhausted ClickHouse retries - **WHEN** three configured insert attempts fail -- **THEN** publish original vendor payloads with ch_insert_failed and commit only after all DLQ deliveries succeed +- **THEN** publish original vendor payloads with `normalization_error` and commit only after all DLQ deliveries succeed +- **THEN** publish original vendor payloads with normalization_error and commit only after all DLQ deliveries succeed ### Requirement: Storage-safe normalization The worker SHALL reject numeric booleans, fractional numeric fields, out-of-range UInt64 and UInt32 values including total bytes, invalid IPv4, and timestamps outside DateTime range without crashing.