feat: containerize api and worker services
This commit is contained in:
38
README.md
38
README.md
@@ -93,6 +93,11 @@ worker processing is not connected yet.
|
||||
* Batch size and in-flight buffer metrics
|
||||
* Kafka consumer lag by partition
|
||||
* Aggregated human-readable `/stats` endpoint
|
||||
* Fully containerized API and processing worker
|
||||
* Health-based Docker Compose startup ordering
|
||||
* Automatic Kafka topic and ClickHouse schema provisioning
|
||||
* Persistent Redpanda and ClickHouse storage
|
||||
* Complete stack startup with `docker compose up --build`
|
||||
|
||||
## Project Structure
|
||||
|
||||
@@ -170,10 +175,10 @@ This includes:
|
||||
* ClickHouse connection settings
|
||||
* Worker batch size and flush interval
|
||||
|
||||
## Running Infrastructure
|
||||
## Running Pipeline
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Verify services:
|
||||
@@ -190,28 +195,23 @@ redpanda-init exited successfully
|
||||
ClickHouse healthy
|
||||
```
|
||||
|
||||
## Running the API
|
||||
### Available endpoints:
|
||||
|
||||
```bash
|
||||
uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Available endpoints:
|
||||
|
||||
API:
|
||||
#### API:
|
||||
|
||||
```text
|
||||
GET /health
|
||||
GET /metrics
|
||||
GET /stats
|
||||
POST /api/v1/cdr/raw
|
||||
GET http://localhost:8000/health
|
||||
GET http://localhost:8000/metrics
|
||||
GET http://localhost:8000/stats
|
||||
POST http://localhost:8000/api/v1/cdr/raw
|
||||
```
|
||||
|
||||
Worker:
|
||||
#### Worker:
|
||||
|
||||
```text
|
||||
GET /metrics
|
||||
GET /internal/stats
|
||||
GET http://localhost:8001/health
|
||||
GET http://localhost:8001/metrics
|
||||
GET http://localhost:8001/internal/stats
|
||||
```
|
||||
|
||||
A valid CDR currently returns:
|
||||
@@ -227,14 +227,12 @@ Invalid payloads return structured `400` responses.
|
||||
|
||||
## Testing
|
||||
|
||||
Run:
|
||||
Once the full stack starts cleanly, a real CDR reaches ClickHouse, and:
|
||||
|
||||
```bash
|
||||
pytest -v
|
||||
```
|
||||
|
||||
The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.
|
||||
|
||||
## Delivery Semantics
|
||||
|
||||
The worker uses at-least-once delivery.
|
||||
|
||||
Reference in New Issue
Block a user