67 lines
4.6 KiB
Markdown
67 lines
4.6 KiB
Markdown
## Purpose
|
|
|
|
Provides a fast, non-blocking HTTP ingestion gate for raw vendor CDRs with backpressure control, publishing to Kafka, and unified operational statistics.
|
|
|
|
## ADDED Requirements
|
|
|
|
### Requirement: Raw CDR Ingestion Endpoint
|
|
The Ingestion API SHALL provide an HTTP endpoint at `POST /api/v1/cdr/raw` to ingest vendor-specific CDR JSON payloads.
|
|
|
|
#### Scenario: Valid CDR payload accepted
|
|
- **WHEN** a client sends a `POST /api/v1/cdr/raw` request with valid JSON containing all required fields (`raw_id`, `device_id`, `caller_num`, `svc`, `rat`, `epoch_start`, `epoch_end`, `up_bytes`, `down_bytes`, `lac`, `ci`) and the internal buffer has capacity
|
|
- **THEN** the API returns HTTP status `202 Accepted` with a structured JSON confirmation only after Kafka acknowledges delivery; pending publications count against the bounded capacity
|
|
|
|
#### Scenario: Malformed JSON payload rejected
|
|
- **WHEN** a client sends a `POST /api/v1/cdr/raw` request with an unparseable or corrupted JSON body
|
|
- **THEN** the API returns HTTP status `400 Bad Request` with structured JSON `{"error": "invalid_json", "detail": "<error_message>"}`
|
|
|
|
#### Scenario: Missing required fields rejected
|
|
- **WHEN** a client sends a `POST /api/v1/cdr/raw` request missing one or more required fields (such as `caller_num` or `raw_id`)
|
|
- **THEN** the API returns HTTP status `400 Bad Request` with structured JSON `{"error": "missing_required_field", "detail": "Field <field_name> is required"}`
|
|
|
|
### Requirement: Bounded Async Queue Backpressure
|
|
The Ingestion API SHALL enforce a bounded in-memory async queue to prevent unbounded memory growth when downstream publishing is delayed.
|
|
|
|
#### Scenario: Internal queue capacity saturated
|
|
- **WHEN** incoming request rate exceeds publishing capacity and the bounded async queue reaches its maximum configured limit
|
|
- **THEN** the API immediately returns HTTP status `429 Too Many Requests` with structured JSON `{"error": "queue_full", "detail": "Ingestion buffer saturated"}` without blocking the event loop
|
|
|
|
### Requirement: Broker Availability Protection
|
|
The Ingestion API SHALL detect Kafka unavailability and return a service unavailable response.
|
|
|
|
#### Scenario: Kafka broker unreachable
|
|
- **WHEN** the Kafka producer fails to connect or queue messages within the configured timeout
|
|
- **THEN** the API returns HTTP status `503 Service Unavailable` with structured JSON `{"error": "broker_unavailable", "detail": "Kafka broker unreachable"}`
|
|
|
|
### Requirement: Partition Keyed Kafka Publishing
|
|
The Ingestion API SHALL publish accepted raw records to the Kafka topic `cdr.raw` using `raw_id` as the message key to preserve record affinity.
|
|
|
|
#### Scenario: Record published with raw_id partition key
|
|
- **WHEN** a valid raw CDR record is queued for dispatch
|
|
- **THEN** the producer publishes the message bytes to `cdr.raw` with Kafka message key set to the record's `raw_id`
|
|
|
|
### Requirement: Prometheus Metrics Endpoint
|
|
The Ingestion API SHALL expose a Prometheus metrics endpoint at `GET /metrics`.
|
|
|
|
#### Scenario: Metrics scraped by Prometheus
|
|
- **WHEN** a Prometheus scraper issues a `GET /metrics` request to the Ingestion API
|
|
- **THEN** the API returns HTTP status `200 OK` with text-based Prometheus metrics including `cdr_api_received_total`
|
|
|
|
### Requirement: Unified Pipeline Stats Endpoint
|
|
The Ingestion API SHALL expose a human-readable JSON status endpoint at `GET /stats` combining local API ingestion counters with runtime metrics retrieved from the Processing Worker.
|
|
|
|
#### Scenario: Unified stats requested
|
|
- **WHEN** an operator issues a `GET /stats` request
|
|
- **THEN** the API queries the Worker's internal stats endpoint, aggregates `total_received` with worker metrics (`total_processed`, `total_dlq`, `in_flight_buffer`, `last_flush_at`, and breakdown maps), and returns HTTP status `200 OK` with structured JSON
|
|
|
|
### Requirement: Acknowledged bounded acceptance and shutdown
|
|
The API SHALL hold admission capacity until Kafka delivery completes, continuously service delivery callbacks, and drain on shutdown.
|
|
|
|
#### Scenario: Failure, timeout or saturation
|
|
- **WHEN** publication fails or exceeds the configured 10-second default deadline, or outstanding capacity is full
|
|
- **THEN** return structured 503 or 429 respectively; never acknowledge an unconfirmed publication, and allow subsequent requests after broker recovery
|
|
|
|
#### Scenario: Invalid types and bounded requests
|
|
- **WHEN** a request contains invalid UTF-8, non-finite JSON, wrong field types, an oversized body, or exceeds the read deadline
|
|
- **THEN** return structured 400, 413, or 408 respectively without publishing; default body/read limits are 65536 bytes and 10 seconds
|