feat: add bounded kafka ingestion pipeline

This commit is contained in:
2026-09-20 14:11:00 +03:30
parent 12cb9b3832
commit 89a746227b
11 changed files with 465 additions and 28 deletions

View File

@@ -21,7 +21,9 @@ FastAPI API
↓
Validation
↓
202 Accepted
Bounded Async Queue
↓
Redpanda / cdr.raw
```
Infrastructure already available:
@@ -35,7 +37,7 @@ ClickHouse
└── telecom.cdr_records
```
Kafka publishing and worker processing are not connected yet.
worker processing is not connected yet.
## Current Features
@@ -53,16 +55,27 @@ Kafka publishing and worker processing are not connected yet.
* FastAPI `GET /health`
* Structured JSON error responses
* Unit and API integration tests
* Bounded asynchronous ingestion queue for backpressure
* Long-lived asynchronous Kafka producer
* Kafka delivery acknowledgement before `202 Accepted`
* `429 Too Many Requests` on ingestion queue saturation
* `503 Service Unavailable` on Kafka delivery failure
## Project Structure
```text
src/core/
├── config.py
│
├── api/
│ ├── __init__.py
│ ├── kafka_publisher.py
│ ├── main.py
│ ├── publisher.py
│ └── routes.py
│
└── domain/
├── __init__.py
├── errors.py
├── models.py
├── normalization.py