44 lines
2.5 KiB
Markdown
44 lines
2.5 KiB
Markdown
## Purpose
|
|
|
|
Defines Prometheus metrics collection, operational stats reporting, and structured JSON logging across pipeline components.
|
|
|
|
## ADDED Requirements
|
|
|
|
### Requirement: Prometheus Metrics Exposition
|
|
Each pipeline service SHALL expose standard Prometheus metrics at `GET /metrics`.
|
|
|
|
#### Scenario: Ingestion API metrics exposed
|
|
- **WHEN** Prometheus scrapes `GET /metrics` on the Ingestion API
|
|
- **THEN** the response includes counter `cdr_api_received_total`
|
|
|
|
#### Scenario: Processing Worker metrics exposed
|
|
- **WHEN** Prometheus scrapes `GET /metrics` on the Processing Worker
|
|
- **THEN** the response includes counters (`cdr_processed_total{service_type, radio_access_type}`, `cdr_dlq_total{reason}`), histograms (`cdr_kafka_to_clickhouse_latency_seconds`, `cdr_batch_flush_duration_seconds`, `cdr_batch_size_at_flush`), and gauges (`cdr_in_flight_buffer_size`, `cdr_kafka_consumer_lag{partition}`)
|
|
|
|
### Requirement: Human-Readable Pipeline Stats JSON
|
|
The Ingestion API SHALL serve a unified JSON summary at `GET /stats` representing end-to-end pipeline health.
|
|
|
|
#### Scenario: Stats endpoint returns unified summary
|
|
- **WHEN** a client issues `GET /stats` to the Ingestion API
|
|
- **THEN** the API returns HTTP 200 with JSON matching the structure:
|
|
- `pipeline`: `total_received`, `total_processed`, `total_dlq`, `in_flight_buffer`, `last_flush_at`
|
|
- `breakdown`: `by_service_type`, `by_radio_access_type`, `dlq_by_reason`
|
|
|
|
### Requirement: Structured JSON Logging
|
|
All pipeline services SHALL emit log lines formatted as structured JSON to stdout.
|
|
|
|
#### Scenario: Operational event logged
|
|
- **WHEN** any service logs an event (info, warning, or error)
|
|
- **THEN** the log output is a single-line JSON object containing at minimum `timestamp` (ISO 8601 UTC), `level` (INFO, WARN, ERROR), `service` (`ingestion-api` or `processing-worker`), `raw_id` (record identifier or null), and `event` (short descriptive string)
|
|
|
|
### Requirement: Measured and truthful telemetry
|
|
Metrics SHALL count acknowledged destination operations, measure Kafka timestamp-to-write latency and periodically update partition lag from committed offsets and high watermarks.
|
|
|
|
#### Scenario: Delivery and scrape
|
|
- **WHEN** records complete storage and metrics are scraped
|
|
- **THEN** latency observations and partition gauges are populated, failed DLQ attempts do not increment counters, and label values are correctly escaped
|
|
|
|
#### Scenario: Failure diagnostics
|
|
- **WHEN** a structured event is logged
|
|
- **THEN** preserve the human-readable message including failure details alongside the event identifier
|