feat: add cdr processing worker and dlq routing

This commit is contained in:
2026-09-20 14:38:05 +03:30
parent 89a746227b
commit 0e23e5bde8
10 changed files with 623 additions and 22 deletions

View File

@@ -24,6 +24,12 @@ Validation
Bounded Async Queue
↓
Redpanda / cdr.raw
↓
Processing Worker
↓
Validation + Normalization
├── valid → processed by worker
└── invalid → cdr.raw.dlq
```
Infrastructure already available:
@@ -60,34 +66,67 @@ worker processing is not connected yet.
* Kafka delivery acknowledgement before `202 Accepted`
* `429 Too Many Requests` on ingestion queue saturation
* `503 Service Unavailable` on Kafka delivery failure
* Kafka consumer with manual offset management
* Worker-side JSON decoding and validation
* Worker-side CDR normalization
* DLQ routing for malformed and unprocessable Kafka records
* DLQ records with failure reason, failed field, timestamp, and worker instance
* Kafka offsets committed only after the current processing outcome succeeds
## Project Structure
```text
src/core/
├── config.py
telecom-cdr-pipeline/
├── infrastructure/
│ └── clickhouse/
│ └── init.sql
│
├── api/
│ ├── __init__.py
│ ├── kafka_publisher.py
│ ├── main.py
│ ├── publisher.py
│ └── routes.py
├── scripts/
│ ├── sample-valid-cdr.json
│ └── sample-invalid-rat.json
│
└── domain/
├── __init__.py
├── errors.py
├── models.py
├── normalization.py
└── validation.py
infrastructure/
└── clickhouse/
└── init.sql
tests/
├── integration/
└── unit/
├── src/
│ └── core/
│ ├── __init__.py
│ ├── config.py
│ │
│ ├── api/
│ │ ├── __init__.py
│ │ ├── kafka_publisher.py
│ │ ├── main.py
│ │ ├── publisher.py
│ │ └── routes.py
│ │
│ ├── domain/
│ │ ├── __init__.py
│ │ ├── errors.py
│ │ ├── models.py
│ │ ├── normalization.py
│ │ └── validation.py
│ │
│ └── worker/
│ ├── __init__.py
│ ├── consumer.py
│ ├── dlq.py
│ ├── main.py
│ └── processor.py
│
├── tests/
│ ├── integration/
│ │ └── test_ingestion_api.py
│ │
│ └── unit/
│ ├── test_dlq.py
│ ├── test_normalization.py
│ ├── test_project_setup.py
│ ├── test_validation.py
│ └── test_worker_processor.py
│
├── .env.example
├── .gitignore
├── docker-compose.yml
├── pyproject.toml
└── README.md
```
## Configuration