feat: add bounded kafka ingestion pipeline
This commit is contained in:
17
README.md
17
README.md
@@ -21,7 +21,9 @@ FastAPI API
|
||||
↓
|
||||
Validation
|
||||
↓
|
||||
202 Accepted
|
||||
Bounded Async Queue
|
||||
↓
|
||||
Redpanda / cdr.raw
|
||||
```
|
||||
|
||||
Infrastructure already available:
|
||||
@@ -35,7 +37,7 @@ ClickHouse
|
||||
└── telecom.cdr_records
|
||||
```
|
||||
|
||||
Kafka publishing and worker processing are not connected yet.
|
||||
worker processing is not connected yet.
|
||||
|
||||
## Current Features
|
||||
|
||||
@@ -53,16 +55,27 @@ Kafka publishing and worker processing are not connected yet.
|
||||
* FastAPI `GET /health`
|
||||
* Structured JSON error responses
|
||||
* Unit and API integration tests
|
||||
* Bounded asynchronous ingestion queue for backpressure
|
||||
* Long-lived asynchronous Kafka producer
|
||||
* Kafka delivery acknowledgement before `202 Accepted`
|
||||
* `429 Too Many Requests` on ingestion queue saturation
|
||||
* `503 Service Unavailable` on Kafka delivery failure
|
||||
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
src/core/
|
||||
├── config.py
|
||||
│
|
||||
├── api/
|
||||
│ ├── __init__.py
|
||||
│ ├── kafka_publisher.py
|
||||
│ ├── main.py
|
||||
│ ├── publisher.py
|
||||
│ └── routes.py
|
||||
│
|
||||
└── domain/
|
||||
├── __init__.py
|
||||
├── errors.py
|
||||
├── models.py
|
||||
├── normalization.py
|
||||
|
||||
Reference in New Issue
Block a user