feat: containerize api and worker services

This commit is contained in:
2026-09-20 17:19:01 +03:30
parent d2109fad09
commit 85d9003e43
8 changed files with 257 additions and 52 deletions

View File

@@ -93,6 +93,11 @@ worker processing is not connected yet.
* Batch size and in-flight buffer metrics
* Kafka consumer lag by partition
* Aggregated human-readable `/stats` endpoint
* Fully containerized API and processing worker
* Health-based Docker Compose startup ordering
* Automatic Kafka topic and ClickHouse schema provisioning
* Persistent Redpanda and ClickHouse storage
* Complete stack startup with `docker compose up --build`
## Project Structure
@@ -170,10 +175,10 @@ This includes:
* ClickHouse connection settings
* Worker batch size and flush interval
## Running Infrastructure
## Running Pipeline
```bash
docker compose up -d
docker compose up -d --build
```
Verify services:
@@ -190,28 +195,23 @@ redpanda-init exited successfully
ClickHouse healthy
```
## Running the API
### Available endpoints:
```bash
uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000
```
Available endpoints:
API:
#### API:
```text
GET /health
GET /metrics
GET /stats
POST /api/v1/cdr/raw
GET http://localhost:8000/health
GET http://localhost:8000/metrics
GET http://localhost:8000/stats
POST http://localhost:8000/api/v1/cdr/raw
```
Worker:
#### Worker:
```text
GET /metrics
GET /internal/stats
GET http://localhost:8001/health
GET http://localhost:8001/metrics
GET http://localhost:8001/internal/stats
```
A valid CDR currently returns:
@@ -227,14 +227,12 @@ Invalid payloads return structured `400` responses.
## Testing
Run:
Once the full stack starts cleanly, a real CDR reaches ClickHouse, and:
```bash
pytest -v
```
The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.
## Delivery Semantics
The worker uses at-least-once delivery.