feat(api): complete Phase 2 - framework-less asyncio Ingestion API with backpressure and metrics
This commit is contained in:
@@ -24,4 +24,3 @@ The storage layer and worker client SHALL store `source_ip` using the ClickHouse
|
||||
#### Scenario: IPv4 string inserted into table
|
||||
- **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
|
||||
- **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types
|
||||
|
||||
|
||||
@@ -53,4 +53,3 @@ The Ingestion API SHALL expose a human-readable JSON status endpoint at `GET /st
|
||||
#### Scenario: Unified stats requested
|
||||
- **WHEN** an operator issues a `GET /stats` request
|
||||
- **THEN** the API queries the Worker's internal stats endpoint, aggregates `total_received` with worker metrics (`total_processed`, `total_dlq`, `in_flight_buffer`, `last_flush_at`, and breakdown maps), and returns HTTP status `200 OK` with structured JSON
|
||||
|
||||
|
||||
@@ -28,4 +28,3 @@ All messages published to `cdr.raw.dlq` SHALL be wrapped in a standardized JSON
|
||||
#### Scenario: Failure reason taxonomy enforced
|
||||
- **WHEN** any record is routed to `cdr.raw.dlq`
|
||||
- **THEN** the `failure_reason` value matches one of: `missing_required_field`, `invalid_rat_value`, `invalid_epoch`, `malformed_json`, or `normalization_error`
|
||||
|
||||
|
||||
@@ -30,4 +30,3 @@ All pipeline services SHALL emit log lines formatted as structured JSON to stdou
|
||||
#### Scenario: Operational event logged
|
||||
- **WHEN** any service logs an event (info, warning, or error)
|
||||
- **THEN** the log output is a single-line JSON object containing at minimum `timestamp` (ISO 8601 UTC), `level` (INFO, WARN, ERROR), `service` (`ingestion-api` or `processing-worker`), `raw_id` (record identifier or null), and `event` (short descriptive string)
|
||||
|
||||
|
||||
@@ -95,4 +95,3 @@ The Processing Worker SHALL handle `SIGTERM` and `SIGINT` signals to flush buffe
|
||||
#### Scenario: SIGTERM received during operation
|
||||
- **WHEN** a `SIGTERM` signal is received by the worker process
|
||||
- **THEN** the worker stops consuming new messages, flushes any remaining in-memory buffered records to ClickHouse, commits the associated Kafka offsets, and exits cleanly with exit code 0
|
||||
|
||||
|
||||
@@ -46,4 +46,3 @@ The test script SHALL inspect ClickHouse, the DLQ, and API metrics after test ex
|
||||
- Records in DLQ
|
||||
- API 400 errors
|
||||
- Duplicate `raw_id`s resolved
|
||||
|
||||
|
||||
Reference in New Issue
Block a user