feat: add pipeline observability and stats

README.md

@@ -86,6 +86,13 @@ worker processing is not connected yet.
 * Whole-batch offset commits with no partial Kafka acknowledgement
 * Replay-safe storage using deterministic UUIDv5 identifiers and `ReplacingMergeTree`
 * Graceful worker shutdown flushes buffered records before exit
+* Separate Prometheus `/metrics` endpoints for the API and worker
+* API accepted-CDR counter
+* Worker processed and DLQ counters
+* Kafka-to-ClickHouse and batch flush latency histograms
+* Batch size and in-flight buffer metrics
+* Kafka consumer lag by partition
+* Aggregated human-readable `/stats` endpoint
 
 ## Project Structure
 
@@ -108,8 +115,10 @@ telecom-cdr-pipeline/
 │       │   ├── __init__.py
 │       │   ├── kafka_publisher.py
 │       │   ├── main.py
+│       │   ├── metrics.py
 │       │   ├── publisher.py
-│       │   └── routes.py
+│       │   ├── routes.py
+│       │   └── stats.py
 │       │
 │       ├── domain/
 │       │   ├── __init__.py
@@ -125,7 +134,10 @@ telecom-cdr-pipeline/
 │           ├── consumer.py
 │           ├── dlq.py
 │           ├── main.py
-│           └── processor.py
+│           ├── metrics.py
+│           ├── observability.py
+│           ├── processor.py
+│           └── stats.py
 │
 ├── tests/
 │   ├── integration/
@@ -137,7 +149,8 @@ telecom-cdr-pipeline/
 │       ├── test_normalization.py
 │       ├── test_project_setup.py
 │       ├── test_validation.py
-│       └── test_worker_processor.py
+│       ├── test_worker_processor.py
+│       └── test_worker_stats.py.py
 │
 ├── .env.example
 ├── .gitignore
@@ -185,11 +198,22 @@ uvicorn core.api.main:app --reload --host 0.0.0.0 --port 8000
 
 Available endpoints:
 
+API:
+
 ```text
 GET  /health
+GET  /metrics
+GET  /stats
 POST /api/v1/cdr/raw
 ```
 
+Worker:
+
+```text
+GET /metrics
+GET /internal/stats
+```
+
 A valid CDR currently returns:
 
 ```json