feat(api): complete Phase 2 - framework-less asyncio Ingestion API with backpressure and metrics
This commit is contained in:
@@ -6,11 +6,11 @@
|
||||
|
||||
## 2. Framework-less Async Ingestion API
|
||||
|
||||
- [ ] 2.1 Implement lightweight HTTP/1.1 server using standard library `asyncio.start_server` with route dispatching for `POST /api/v1/cdr/raw`, `GET /metrics`, and `GET /stats`.
|
||||
- [ ] 2.2 Implement bounded async queue (`asyncio.Queue`) backpressure: verify `429 Too Many Requests` when queue is saturated and `202 Accepted` when enqueued.
|
||||
- [ ] 2.3 Implement incoming payload validation: verify `400 Bad Request` with structured JSON error `{ "error": "...", "detail": "..." }` on malformed JSON or missing required fields.
|
||||
- [ ] 2.4 Integrate `confluent-kafka` producer publishing to `cdr.raw` using `raw_id` as message key, and return `503 Service Unavailable` if Kafka is unreachable.
|
||||
- [ ] 2.5 Implement Prometheus metrics endpoint (`cdr_api_received_total`) and public `GET /stats` endpoint that fetches worker runtime stats via internal HTTP and returns unified JSON.
|
||||
- [x] 2.1 Implement lightweight HTTP/1.1 server using standard library `asyncio.start_server` with route dispatching for `POST /api/v1/cdr/raw`, `GET /metrics`, and `GET /stats`.
|
||||
- [x] 2.2 Implement bounded async queue (`asyncio.Queue`) backpressure: verify `429 Too Many Requests` when queue is saturated and `202 Accepted` when enqueued.
|
||||
- [x] 2.3 Implement incoming payload validation: verify `400 Bad Request` with structured JSON error `{ "error": "...", "detail": "..." }` on malformed JSON or missing required fields.
|
||||
- [x] 2.4 Integrate `confluent-kafka` producer publishing to `cdr.raw` using `raw_id` as message key, and return `503 Service Unavailable` if Kafka is unreachable.
|
||||
- [x] 2.5 Implement Prometheus metrics endpoint (`cdr_api_received_total`) and public `GET /stats` endpoint that fetches worker runtime stats via internal HTTP and returns unified JSON.
|
||||
|
||||
## 3. Processing Worker Implementation
|
||||
|
||||
@@ -32,4 +32,3 @@
|
||||
- [ ] 5.1 Implement traffic generator script covering the 7 mandatory scenarios (valid records, missing fields, malformed JSON, bad RAT, invalid epoch, duplicates, burst load).
|
||||
- [ ] 5.2 Implement automated verification summary reporting verifying ClickHouse counts, DLQ counts, API 400 errors, and duplicate resolution.
|
||||
- [ ] 5.3 Write comprehensive `README.md` covering quickstart, architecture, micro-batching, offset management, deduplication, monitoring, and configuration reference.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user