feat(load-test): replace k6 with locust for ongoing continuous testing and web GUI

README.md

@@ -99,6 +99,22 @@ worker processing is not connected yet.
 * Persistent Redpanda and ClickHouse storage
 * Complete stack startup with `docker compose up --build`
 
+## Observability Architecture
+
+Observability is strictly service-owned; each service manages its own telemetry and monitoring components:
+
+* **API Observability** (`core.api.observability`):
+  * Prometheus metrics: `cdr_api_received_total` counter.
+  * API runtime stats: In-memory counter for accepted CDR requests.
+  * Worker stats HTTP client: `HttpWorkerStatsClient` implementing `WorkerStatsProvider` to query the worker's internal stats endpoint.
+  * Observability routes: Dedicated router mounted on the API for `GET /metrics` and `GET /stats`.
+  * The `GET /stats` endpoint aggregates API reception stats with worker processing stats retrieved over HTTP.
+
+* **Worker Observability** (`core.worker.observability`):
+  * Prometheus metrics: Processed and DLQ counters, batch flush duration and Kafka-to-ClickHouse latency histograms, in-flight buffer and consumer lag gauges.
+  * Runtime stats: Thread-safe in-memory breakdown by service type, radio access type, DLQ failure reason, and buffer status.
+  * Internal observability HTTP server: `WorkerObservabilityServer` running a background ASGI server providing `GET /health`, `GET /metrics`, and `GET /internal/stats`.
+
 ## Project Structure
 
 ```text
@@ -108,8 +124,9 @@ telecom-cdr-pipeline/
 │       └── init.sql
 │
 ├── scripts/
-│   ├── sample-valid-cdr.json
-│   └── sample-invalid-rat.json
+│   ├── locustfile.py
+│   ├── functional_traffic.py
+│   └── sample-valid-cdr.json
 │
 ├── src/
 │   └── core/
@@ -120,10 +137,13 @@ telecom-cdr-pipeline/
 │       │   ├── __init__.py
 │       │   ├── kafka_publisher.py
 │       │   ├── main.py
-│       │   ├── metrics.py
 │       │   ├── publisher.py
 │       │   ├── routes.py
-│       │   └── stats.py
+│       │   └── observability/
+│       │       ├── __init__.py
+│       │       ├── metrics.py
+│       │       ├── routes.py
+│       │       └── stats.py
 │       │
 │       ├── domain/
 │       │   ├── __init__.py
@@ -139,23 +159,27 @@ telecom-cdr-pipeline/
 │           ├── consumer.py
 │           ├── dlq.py
 │           ├── main.py
-│           ├── metrics.py
-│           ├── observability.py
 │           ├── processor.py
-│           └── stats.py
+│           └── observability/
+│               ├── __init__.py
+│               ├── metrics.py
+│               ├── server.py
+│               └── stats.py
 │
 ├── tests/
 │   ├── integration/
 │   │   └── test_ingestion_api.py
 │   │
 │   └── unit/
+│       ├── test_api_observability.py
 │       ├── test_batch.py
 │       ├── test_dlq.py
 │       ├── test_normalization.py
 │       ├── test_project_setup.py
 │       ├── test_validation.py
+│       ├── test_worker_observability.py
 │       ├── test_worker_processor.py
-│       └── test_worker_stats.py.py
+│       └── test_worker_stats.py
 │
 ├── .env.example
 ├── .gitignore
@@ -233,6 +257,71 @@ Once the full stack starts cleanly, a real CDR reaches ClickHouse, and:
 pytest -v
 ```
 
+## Load Testing with Locust (Ongoing Test & Web GUI)
+
+A high-throughput Locust load testing suite is provided in `scripts/locustfile.py` using `FastHttpUser` (`geventhttpclient`). It runs an **ongoing test that executes continuously until you stop it** and exposes an interactive **Web GUI** to monitor real-time throughput, latency distributions, and backpressure.
+
+### Option 1: Run Locust via Local Python
+
+1. Ensure dependencies are installed:
+   ```bash
+   pip install -e ".[load-test]"
+   ```
+2. Start the Locust Web GUI:
+   ```bash
+   locust -f scripts/locustfile.py --host http://localhost:8000
+   ```
+3. Open your browser at **http://localhost:8089**.
+
+### Option 2: Run Locust via Docker Compose
+
+Locust is configured as an on-demand service under the `load-test` profile in `docker-compose.yml`:
+
+```bash
+docker compose --profile load-test up locust
+```
+
+Open your browser at **http://localhost:8089**. The target host is pre-configured to `http://api:8000`.
+
+---
+
+### Controlling the Ongoing Test from the Web GUI
+
+1. Open **http://localhost:8089**.
+2. Configure your test parameters:
+   - **Number of users**: e.g., `50` (or `500` for high stress).
+   - **Spawn rate**: e.g., `10` users per second.
+   - **Host**: `http://localhost:8000` (or `http://api:8000` inside Docker).
+3. Click **Start swarming**:
+   - The test will **run continuously and never stop** automatically.
+   - You can inspect real-time metrics across tabs:
+     - **Statistics**: Total requests, RPS, 50th/90th/95th/99th percentiles, and failure counts.
+     - **Charts**: Live charts for Total Requests per Second, Response Times (ms), and Number of Active Users.
+     - **Failures**: Distinguishes normal graceful backpressure (`429 Too Many Requests`) from broker errors (`503 Service Unavailable`).
+     - **Download Data**: Export requests/failures CSVs or generate full HTML test reports.
+4. Adjust or Stop:
+   - You can change the user count in real-time from the GUI top bar.
+   - Click **Stop swarming** whenever you want to end the test.
+
+---
+
+### Correlating with Pipeline Observability
+
+While the Locust test is running ongoing traffic:
+
+1. Check API metrics:
+   ```bash
+   curl -s http://localhost:8000/metrics | grep cdr_api_received_total
+   ```
+2. Check worker consumer lag:
+   ```bash
+   curl -s http://localhost:8001/metrics | grep cdr_kafka_consumer_lag
+   ```
+3. Check aggregated pipeline statistics:
+   ```bash
+   curl -s http://localhost:8000/stats
+   ```
+
 ## Delivery Semantics
 
 The worker uses at-least-once delivery.