feat(api): complete Phase 2 - framework-less asyncio Ingestion API with backpressure and metrics

openspec/changes/telecom-cdr-pipeline/design.md

@@ -69,4 +69,3 @@ The architecture is constrained to pure Python without web frameworks (FastAPI,
 - **[Risk] ClickHouse outage stalls worker** → **[Mitigation]** Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed to prevent infinite consumer lag loops.
 - **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`.
 - **[Risk] Phone number parsing performance overhead** → **[Mitigation]** Use `phonenumbers` library with fast-path prefix checks (`00`, `011`, `+`) before full parsing.
-

openspec/changes/telecom-cdr-pipeline/proposal.md

@@ -30,4 +30,3 @@ Telecom network elements generate high-volume, heterogeneous Call Detail Records
 - **Dependencies**: `confluent-kafka`, `clickhouse-connect`, `phonenumbers` (no web frameworks).
 - **Infrastructure**: Kafka/Redpanda broker and ClickHouse database managed via Docker Compose.
 - **Data Flow**: Network Element → HTTP API → Kafka (`cdr.raw`) → Worker → ClickHouse (`cdr_records`) / DLQ (`cdr.raw.dlq`).
-

openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md

@@ -24,4 +24,3 @@ The storage layer and worker client SHALL store `source_ip` using the ClickHouse
 #### Scenario: IPv4 string inserted into table
 - **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
 - **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types
-

openspec/changes/telecom-cdr-pipeline/specs/ingestion-api/spec.md

@@ -53,4 +53,3 @@ The Ingestion API SHALL expose a human-readable JSON status endpoint at `GET /st
 #### Scenario: Unified stats requested
 - **WHEN** an operator issues a `GET /stats` request
 - **THEN** the API queries the Worker's internal stats endpoint, aggregates `total_received` with worker metrics (`total_processed`, `total_dlq`, `in_flight_buffer`, `last_flush_at`, and breakdown maps), and returns HTTP status `200 OK` with structured JSON
-

openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md

@@ -28,4 +28,3 @@ All messages published to `cdr.raw.dlq` SHALL be wrapped in a standardized JSON
 #### Scenario: Failure reason taxonomy enforced
 - **WHEN** any record is routed to `cdr.raw.dlq`
 - **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, or `normalization_error`
-

openspec/changes/telecom-cdr-pipeline/specs/observability/spec.md

@@ -30,4 +30,3 @@ All pipeline services SHALL emit log lines formatted as structured JSON to stdou
 #### Scenario: Operational event logged
 - **WHEN** any service logs an event (info, warning, or error)
 - **THEN** the log output is a single-line JSON object containing at minimum `timestamp` (ISO 8601 UTC), `level` (INFO, WARN, ERROR), `service` (`ingestion-api` or `processing-worker`), `raw_id` (record identifier or null), and `event` (short descriptive string)
-

openspec/changes/telecom-cdr-pipeline/specs/processing-worker/spec.md

@@ -95,4 +95,3 @@ The Processing Worker SHALL handle `SIGTERM` and `SIGINT` signals to flush buffe
 #### Scenario: SIGTERM received during operation
 - **WHEN** a `SIGTERM` signal is received by the worker process
 - **THEN** the worker stops consuming new messages, flushes any remaining in-memory buffered records to ClickHouse, commits the associated Kafka offsets, and exits cleanly with exit code 0
-

openspec/changes/telecom-cdr-pipeline/specs/verification/spec.md

@@ -46,4 +46,3 @@ The test script SHALL inspect ClickHouse, the DLQ, and API metrics after test ex
   - Records in DLQ
   - API 400 errors
   - Duplicate `raw_id`s resolved
-

openspec/changes/telecom-cdr-pipeline/tasks.md

@@ -6,11 +6,11 @@
 
 ## 2. Framework-less Async Ingestion API
 
-- [ ] 2.1 Implement lightweight HTTP/1.1 server using standard library `asyncio.start_server` with route dispatching for `POST /api/v1/cdr/raw`, `GET /metrics`, and `GET /stats`.
-- [ ] 2.2 Implement bounded async queue (`asyncio.Queue`) backpressure: verify `429 Too Many Requests` when queue is saturated and `202 Accepted` when enqueued.
-- [ ] 2.3 Implement incoming payload validation: verify `400 Bad Request` with structured JSON error `{ "error": "...", "detail": "..." }` on malformed JSON or missing required fields.
-- [ ] 2.4 Integrate `confluent-kafka` producer publishing to `cdr.raw` using `raw_id` as message key, and return `503 Service Unavailable` if Kafka is unreachable.
-- [ ] 2.5 Implement Prometheus metrics endpoint (`cdr_api_received_total`) and public `GET /stats` endpoint that fetches worker runtime stats via internal HTTP and returns unified JSON.
+- [x] 2.1 Implement lightweight HTTP/1.1 server using standard library `asyncio.start_server` with route dispatching for `POST /api/v1/cdr/raw`, `GET /metrics`, and `GET /stats`.
+- [x] 2.2 Implement bounded async queue (`asyncio.Queue`) backpressure: verify `429 Too Many Requests` when queue is saturated and `202 Accepted` when enqueued.
+- [x] 2.3 Implement incoming payload validation: verify `400 Bad Request` with structured JSON error `{ "error": "...", "detail": "..." }` on malformed JSON or missing required fields.
+- [x] 2.4 Integrate `confluent-kafka` producer publishing to `cdr.raw` using `raw_id` as message key, and return `503 Service Unavailable` if Kafka is unreachable.
+- [x] 2.5 Implement Prometheus metrics endpoint (`cdr_api_received_total`) and public `GET /stats` endpoint that fetches worker runtime stats via internal HTTP and returns unified JSON.
 
 ## 3. Processing Worker Implementation
 
@@ -32,4 +32,3 @@
 - [ ] 5.1 Implement traffic generator script covering the 7 mandatory scenarios (valid records, missing fields, malformed JSON, bad RAT, invalid epoch, duplicates, burst load).
 - [ ] 5.2 Implement automated verification summary reporting verifying ClickHouse counts, DLQ counts, API 400 errors, and duplicate resolution.
 - [ ] 5.3 Write comprehensive `README.md` covering quickstart, architecture, micro-batching, offset management, deduplication, monitoring, and configuration reference.
-