telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/specs/ingestion-api/spec.md

Purpose

Provides a fast, non-blocking HTTP ingestion gate for raw vendor CDRs with backpressure control, publishing to Kafka, and unified operational statistics.

ADDED Requirements

Requirement: Raw CDR Ingestion Endpoint

The Ingestion API SHALL provide an HTTP endpoint at POST /api/v1/cdr/raw to ingest vendor-specific CDR JSON payloads.

Scenario: Valid CDR payload accepted

• WHEN a client sends a POST /api/v1/cdr/raw request with valid JSON containing all required fields (raw_id, device_id, caller_num, svc, rat, epoch_start, epoch_end, up_bytes, down_bytes, lac, ci) and the internal buffer has capacity
• THEN the API returns HTTP status 202 Accepted with a structured JSON confirmation and queues the payload for Kafka publishing

Scenario: Malformed JSON payload rejected

• WHEN a client sends a POST /api/v1/cdr/raw request with an unparseable or corrupted JSON body
• THEN the API returns HTTP status 400 Bad Request with structured JSON {"error": "malformed_json", "detail": "<error_message>"}

Scenario: Missing required fields rejected

• WHEN a client sends a POST /api/v1/cdr/raw request missing one or more required fields (such as caller_num or raw_id)
• THEN the API returns HTTP status 400 Bad Request with structured JSON {"error": "missing_required_field", "detail": "Field <field_name> is required"}

Requirement: Bounded Async Queue Backpressure

The Ingestion API SHALL enforce a bounded in-memory async queue to prevent unbounded memory growth when downstream publishing is delayed.

Scenario: Internal queue capacity saturated

• WHEN incoming request rate exceeds publishing capacity and the bounded async queue reaches its maximum configured limit
• THEN the API immediately returns HTTP status 429 Too Many Requests with structured JSON {"error": "queue_full", "detail": "Ingestion buffer saturated"} without blocking the event loop

Requirement: Broker Availability Protection

The Ingestion API SHALL detect Kafka unavailability and return a service unavailable response.

Scenario: Kafka broker unreachable

• WHEN the Kafka producer fails to connect or queue messages within the configured timeout
• THEN the API returns HTTP status 503 Service Unavailable with structured JSON {"error": "broker_unavailable", "detail": "Kafka broker unreachable"}

Requirement: Partition Keyed Kafka Publishing

The Ingestion API SHALL publish accepted raw records to the Kafka topic cdr.raw using raw_id as the message key to preserve record affinity.

Scenario: Record published with raw_id partition key

• WHEN a valid raw CDR record is queued for dispatch
• THEN the producer publishes the message bytes to cdr.raw with Kafka message key set to the record's raw_id

Requirement: Prometheus Metrics Endpoint

The Ingestion API SHALL expose a Prometheus metrics endpoint at GET /metrics.

Scenario: Metrics scraped by Prometheus

• WHEN a Prometheus scraper issues a GET /metrics request to the Ingestion API
• THEN the API returns HTTP status 200 OK with text-based Prometheus metrics including cdr_api_received_total

Requirement: Unified Pipeline Stats Endpoint

The Ingestion API SHALL expose a human-readable JSON status endpoint at GET /stats combining local API ingestion counters with runtime metrics retrieved from the Processing Worker.

Scenario: Unified stats requested

• WHEN an operator issues a GET /stats request
• THEN the API queries the Worker's internal stats endpoint, aggregates total_received with worker metrics (total_processed, total_dlq, in_flight_buffer, last_flush_at, and breakdown maps), and returns HTTP status 200 OK with structured JSON