diff --git a/README.md b/README.md index cfe4985..a569c75 100644 --- a/README.md +++ b/README.md @@ -4,19 +4,140 @@ High-throughput telecom Call Detail Record ingestion and analytics pipeline. ## Architecture -The system will consist of: +The system consists of: -1. FastAPI ingestion API -2. Kafka-compatible Redpanda message broker -3. Processing and normalization worker -4. ClickHouse analytics database +* FastAPI ingestion API +* Redpanda as a Kafka-compatible message broker +* Processing and normalization worker +* ClickHouse analytical storage +* Dead Letter Queue for failed records -Data flow: +Current data flow: -Network Element → Ingestion API → Redpanda → Worker → ClickHouse +```text +Network Element + ↓ +FastAPI API + ↓ +Validation + ↓ +202 Accepted +``` -Malformed or unprocessable records are routed by the worker to a Dead Letter Queue. +Infrastructure already available: -## Development Status +```text +Redpanda +├── cdr.raw +└── cdr.raw.dlq -Currently under development. \ No newline at end of file +ClickHouse +└── telecom.cdr_records +``` + +Kafka publishing and worker processing are not connected yet. + +## Current Features + +* Dockerized Redpanda and ClickHouse +* Automatic and idempotent Kafka topic provisioning +* Automatic ClickHouse schema initialization +* Persistent Redpanda and ClickHouse volumes +* Raw and normalized CDR domain models +* Required-field and type validation +* RAT validation and normalization +* E.164 phone-number normalization +* Deterministic UUIDv5 `cdr_id` generation +* Epoch and traffic-value validation +* FastAPI `POST /api/v1/cdr/raw` +* FastAPI `GET /health` +* Structured JSON error responses +* Unit and API integration tests + +## Project Structure + +```text +src/core/ +├── api/ +│ ├── main.py +│ └── routes.py +│ +└── domain/ + ├── errors.py + ├── models.py + ├── normalization.py + └── validation.py + +infrastructure/ +└── clickhouse/ + └── init.sql + +tests/ +├── integration/ +└── unit/ +``` + +## Configuration + +Configuration is provided through environment variables documented in `.env.example`. + +This includes: + +* Redpanda version and topic settings +* Kafka broker and consumer configuration +* ClickHouse connection settings +* Worker batch size and flush interval + +## Running Infrastructure + +```bash +docker compose up -d +``` + +Verify services: + +```bash +docker compose ps -a +``` + +Expected: + +```text +Redpanda healthy +redpanda-init exited successfully +ClickHouse healthy +``` + +## Running the API + +```bash +uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000 +``` + +Available endpoints: + +```text +GET /health +POST /api/v1/cdr/raw +``` + +A valid CDR currently returns: + +```json +{ + "status": "accepted", + "raw_id": "raw-test-001" +} +``` + +Invalid payloads return structured `400` responses. + +## Testing + +Run: + +```bash +pytest -v +``` + +The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation. diff --git a/pyproject.toml b/pyproject.toml index d89d360..6fd6330 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,7 +12,7 @@ dependencies = [ [project.optional-dependencies] dev = [ "pytest>=8.0", - "httpx>=0.27,<1.0", + "httpx2>=2.4,<3.0", ] [build-system] diff --git a/src/core/api/main.py b/src/core/api/main.py index 4884f65..872da07 100644 --- a/src/core/api/main.py +++ b/src/core/api/main.py @@ -19,4 +19,4 @@ app.include_router( async def health() -> dict[str, str]: return { "status": "ok", - } \ No newline at end of file + }