telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/specs/observability/spec.md

Purpose

Defines Prometheus metrics collection, operational stats reporting, and structured JSON logging across pipeline components.

ADDED Requirements

Requirement: Prometheus Metrics Exposition

Each pipeline service SHALL expose standard Prometheus metrics at GET /metrics.

Scenario: Ingestion API metrics exposed

• WHEN Prometheus scrapes GET /metrics on the Ingestion API
• THEN the response includes counter cdr_api_received_total

Scenario: Processing Worker metrics exposed

• WHEN Prometheus scrapes GET /metrics on the Processing Worker
• THEN the response includes counters (cdr_processed_total{service_type, radio_access_type}, cdr_dlq_total{reason}), histograms (cdr_kafka_to_clickhouse_latency_seconds, cdr_batch_flush_duration_seconds, cdr_batch_size_at_flush), and gauges (cdr_in_flight_buffer_size, cdr_kafka_consumer_lag{partition})

Requirement: Human-Readable Pipeline Stats JSON

The Ingestion API SHALL serve a unified JSON summary at GET /stats representing end-to-end pipeline health.

Scenario: Stats endpoint returns unified summary

• WHEN a client issues GET /stats to the Ingestion API
• THEN the API returns HTTP 200 with JSON matching the structure:
  • pipeline: total_received, total_processed, total_dlq, in_flight_buffer, last_flush_at
  • breakdown: by_service_type, by_radio_access_type, dlq_by_reason

Requirement: Structured JSON Logging

All pipeline services SHALL emit log lines formatted as structured JSON to stdout.

Scenario: Operational event logged

• WHEN any service logs an event (info, warning, or error)
• THEN the log output is a single-line JSON object containing at minimum timestamp (ISO 8601 UTC), level (INFO, WARN, ERROR), service (ingestion-api or processing-worker), raw_id (record identifier or null), and event (short descriptive string)

Requirement: Measured and truthful telemetry

Metrics SHALL count acknowledged destination operations, measure Kafka timestamp-to-write latency and periodically update partition lag from committed offsets and high watermarks.

Scenario: Delivery and scrape

• WHEN records complete storage and metrics are scraped
• THEN latency observations and partition gauges are populated, failed DLQ attempts do not increment counters, and label values are correctly escaped

Scenario: Failure diagnostics

• WHEN a structured event is logged
• THEN preserve the human-readable message including failure details alongside the event identifier