docs(pipeline): align delivery contracts and verification requirements

openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md

@@ -9,7 +9,7 @@ The storage layer SHALL provide a database named `telecom` and a table named `cd
 
 #### Scenario: Database and table initialized idempotently
 - **WHEN** the initialization script or container entrypoint executes
-- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
+- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
 
 ### Requirement: Deduplication on At-Least-Once Replay
 The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.
@@ -24,3 +24,14 @@ The storage layer and worker client SHALL store `source_ip` using the ClickHouse
 #### Scenario: IPv4 string inserted into table
 - **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
 - **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types
+
+### Requirement: Identity preservation and schema compatibility
+Initialization SHALL use one canonical schema, verify existing table compatibility and never reset data automatically.
+
+#### Scenario: Distinct identities share analytical metadata
+- **WHEN** records have different raw_id values but identical other sorting fields
+- **THEN** both survive FINAL and background replacement, while exact replay yields one logical row
+
+#### Scenario: Existing incompatible table
+- **WHEN** an existing table lacks the corrected identity sorting key
+- **THEN** initialization fails with explicit assessment reset instructions without deleting data

openspec/changes/telecom-cdr-pipeline/specs/ingestion-api/spec.md

@@ -9,11 +9,11 @@ The Ingestion API SHALL provide an HTTP endpoint at `POST /api/v1/cdr/raw` to in
 
 #### Scenario: Valid CDR payload accepted
 - **WHEN** a client sends a `POST /api/v1/cdr/raw` request with valid JSON containing all required fields (`raw_id`, `device_id`, `caller_num`, `svc`, `rat`, `epoch_start`, `epoch_end`, `up_bytes`, `down_bytes`, `lac`, `ci`) and the internal buffer has capacity
-- **THEN** the API returns HTTP status `202 Accepted` with a structured JSON confirmation and queues the payload for Kafka publishing
+- **THEN** the API returns HTTP status `202 Accepted` with a structured JSON confirmation only after Kafka acknowledges delivery; pending publications count against the bounded capacity
 
 #### Scenario: Malformed JSON payload rejected
 - **WHEN** a client sends a `POST /api/v1/cdr/raw` request with an unparseable or corrupted JSON body
-- **THEN** the API returns HTTP status `400 Bad Request` with structured JSON `{"error": "malformed_json", "detail": "<error_message>"}`
+- **THEN** the API returns HTTP status `400 Bad Request` with structured JSON `{"error": "invalid_json", "detail": "<error_message>"}`
 
 #### Scenario: Missing required fields rejected
 - **WHEN** a client sends a `POST /api/v1/cdr/raw` request missing one or more required fields (such as `caller_num` or `raw_id`)
@@ -53,3 +53,14 @@ The Ingestion API SHALL expose a human-readable JSON status endpoint at `GET /st
 #### Scenario: Unified stats requested
 - **WHEN** an operator issues a `GET /stats` request
 - **THEN** the API queries the Worker's internal stats endpoint, aggregates `total_received` with worker metrics (`total_processed`, `total_dlq`, `in_flight_buffer`, `last_flush_at`, and breakdown maps), and returns HTTP status `200 OK` with structured JSON
+
+### Requirement: Acknowledged bounded acceptance and shutdown
+The API SHALL hold admission capacity until Kafka delivery completes, continuously service delivery callbacks, and drain on shutdown.
+
+#### Scenario: Failure, timeout or saturation
+- **WHEN** publication fails or exceeds the configured 10-second default deadline, or outstanding capacity is full
+- **THEN** return structured 503 or 429 respectively; never acknowledge an unconfirmed publication, and allow subsequent requests after broker recovery
+
+#### Scenario: Invalid types and bounded requests
+- **WHEN** a request contains invalid UTF-8, non-finite JSON, wrong field types, an oversized body, or exceeds the read deadline
+- **THEN** return structured 400, 413, or 408 respectively without publishing; default body/read limits are 65536 bytes and 10 seconds

openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md

@@ -27,4 +27,4 @@ All messages published to `cdr.raw.dlq` SHALL be wrapped in a standardized JSON
 
 #### Scenario: Failure reason taxonomy enforced
 - **WHEN** any record is routed to `cdr.raw.dlq`
-- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, or `normalization_error`
+- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, `normalization_error`, or `ch_insert_failed` (a documented assessment extension for exhausted storage retries)

openspec/changes/telecom-cdr-pipeline/specs/observability/spec.md

@@ -30,3 +30,14 @@ All pipeline services SHALL emit log lines formatted as structured JSON to stdou
 #### Scenario: Operational event logged
 - **WHEN** any service logs an event (info, warning, or error)
 - **THEN** the log output is a single-line JSON object containing at minimum `timestamp` (ISO 8601 UTC), `level` (INFO, WARN, ERROR), `service` (`ingestion-api` or `processing-worker`), `raw_id` (record identifier or null), and `event` (short descriptive string)
+
+### Requirement: Measured and truthful telemetry
+Metrics SHALL count acknowledged destination operations, measure Kafka timestamp-to-write latency and periodically update partition lag from committed offsets and high watermarks.
+
+#### Scenario: Delivery and scrape
+- **WHEN** records complete storage and metrics are scraped
+- **THEN** latency observations and partition gauges are populated, failed DLQ attempts do not increment counters, and label values are correctly escaped
+
+#### Scenario: Failure diagnostics
+- **WHEN** a structured event is logged
+- **THEN** preserve the human-readable message including failure details alongside the event identifier

openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md

@@ -95,3 +95,25 @@ The Processing Worker SHALL handle `SIGTERM` and `SIGINT` signals to flush buffe
 #### Scenario: SIGTERM received during operation
 - **WHEN** a `SIGTERM` signal is received by the worker process
 - **THEN** the worker stops consuming new messages, flushes any remaining in-memory buffered records to ClickHouse, commits the associated Kafka offsets, and exits cleanly with exit code 0
+
+### Requirement: Safe partition progress and confirmed DLQ
+The worker SHALL disable automatic offset storage, commit explicit next offsets, and await all destination acknowledgements.
+
+#### Scenario: Valid record precedes an invalid record
+- **WHEN** an invalid record follows buffered valid records
+- **THEN** flush and commit those valid records before acknowledging and committing the invalid record to DLQ
+
+#### Scenario: Destination or commit failure
+- **WHEN** DLQ delivery or any partition commit fails
+- **THEN** stop nonzero without advancing unsafe offsets or reclassifying the error as a ClickHouse failure; fatal cleanup does not flush the buffer
+
+#### Scenario: Exhausted ClickHouse retries
+- **WHEN** three configured insert attempts fail
+- **THEN** publish original vendor payloads with ch_insert_failed and commit only after all DLQ deliveries succeed
+
+### Requirement: Storage-safe normalization
+The worker SHALL reject numeric booleans, fractional numeric fields, out-of-range UInt64 and UInt32 values including total bytes, invalid IPv4, and timestamps outside DateTime range without crashing.
+
+#### Scenario: Invalid scalar or conversion
+- **WHEN** any field fails its type, range or conversion check
+- **THEN** route only that record to DLQ with a field-specific reason and preserve other valid records

openspec/changes/telecom-cdr-pipeline/specs/verification/spec.md

@@ -13,7 +13,7 @@ The test suite SHALL provide a script that generates synthetic CDR traffic to ex
 
 #### Scenario: Missing required field validation
 - **WHEN** the generator sends 20 records missing `caller_num`
-- **THEN** all 20 records are rejected by the API with HTTP 400 or routed to DLQ with `missing_required_field`
+- **THEN** all 20 records are rejected by the API with HTTP 400; 20 additional direct Kafka injections verify DLQ `missing_required_field`
 
 #### Scenario: Malformed JSON validation
 - **WHEN** the generator sends 10 invalid JSON payloads
@@ -46,3 +46,10 @@ The test script SHALL inspect ClickHouse, the DLQ, and API metrics after test ex
   - Records in DLQ
   - API 400 errors
   - Duplicate `raw_id`s resolved
+
+### Requirement: Independent durable evidence
+Verification SHALL inspect actual run-correlated DLQ envelopes and FINAL ClickHouse rows instead of inferring delivery from counters.
+
+#### Scenario: Full assessment run
+- **WHEN** 1570 HTTP requests and 30 direct Kafka fault injections finish
+- **THEN** verify 1540 HTTP acknowledgements, 30 structured HTTP rejections, 1500 unique stored rows, 50 actual DLQ records, correct transformations and metrics, and a 500-record burst under five seconds; unavailable evidence is failure