amiirkhl/telecom-cdr-pipeline-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(pipeline): align worker and storage contracts

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-09-20 10:40:58 +03:30
parent 81cef7f732
commit dc66e7fe97
4 changed files with 13 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md
View File

@@ -16,7 +16,7 @@ The storage layer SHALL support idempotent re-insertion of records without creat
#### Scenario: Duplicate record inserted after crash replay
- **WHEN** a batch containing previously written records is re-inserted due to consumer restart before offset commit
- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates rows sharing the same order key upon merge or when queried with `FINAL`
- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates exact replays sharing the deterministic `cdr_id`-bearing order key upon merge or when queried with `FINAL`, without collapsing distinct `raw_id` values whose session metadata happens to match
### Requirement: Source IP Column Type Compatibility
The storage layer and worker client SHALL store `source_ip` using the ClickHouse `IPv4` data type.
@@ -29,9 +29,9 @@ The storage layer and worker client SHALL store `source_ip` using the ClickHouse
Initialization SHALL use one canonical schema, verify existing table compatibility and never reset data automatically.
#### Scenario: Distinct identities share analytical metadata
- **WHEN** records have different raw_id values but identical other sorting fields
- **THEN** both survive FINAL and background replacement, while exact replay yields one logical row
- **WHEN** records have different `raw_id` values but identical other sorting fields
- **THEN** both survive `FINAL` and background replacement, while an exact replay yields one logical row
#### Scenario: Existing incompatible table
- **WHEN** an existing table lacks the corrected identity sorting key
- **WHEN** an existing table does not use the canonical identity-bearing sorting key
- **THEN** initialization fails with explicit assessment reset instructions without deleting data

3
openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md
View File

@@ -27,4 +27,5 @@ All messages published to `cdr.raw.dlq` SHALL be wrapped in a standardized JSON
#### Scenario: Failure reason taxonomy enforced
- **WHEN** any record is routed to `cdr.raw.dlq`
- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`, or `ch_insert_failed` (a documented assessment extension for exhausted storage retries)
- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, or `normalization_error`
- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`

4
openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md
View File

@@ -102,6 +102,7 @@ The worker SHALL disable automatic offset storage, commit explicit next offsets,
#### Scenario: Valid record precedes an invalid record
- **WHEN** an invalid record follows buffered valid records
- **THEN** flush and commit those valid records before acknowledging and committing the invalid record to DLQ
- **THEN** route the invalid record to DLQ and defer committing its offset until buffered valid records are flushed, or commit immediately if no preceding uncommitted records exist on that partition
#### Scenario: Destination or commit failure
- **WHEN** DLQ delivery or any partition commit fails
@@ -109,7 +110,8 @@ The worker SHALL disable automatic offset storage, commit explicit next offsets,
#### Scenario: Exhausted ClickHouse retries
- **WHEN** three configured insert attempts fail
- **THEN** publish original vendor payloads with ch_insert_failed and commit only after all DLQ deliveries succeed
- **THEN** publish original vendor payloads with `normalization_error` and commit only after all DLQ deliveries succeed
- **THEN** publish original vendor payloads with normalization_error and commit only after all DLQ deliveries succeed
### Requirement: Storage-safe normalization
The worker SHALL reject numeric booleans, fractional numeric fields, out-of-range UInt64 and UInt32 values including total bytes, invalid IPv4, and timestamps outside DateTime range without crashing.