telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/specs/message-broker/spec.md

Purpose

Defines message broker topics, partitioning, retention, and dead-letter queue envelope specifications for decoupling and reliable asynchronous buffer management.

ADDED Requirements

Requirement: Raw Ingestion Topic Configuration

The message broker SHALL provide a topic named cdr.raw configured for high-throughput buffering of raw vendor CDR payloads.

Scenario: Raw topic configuration applied

• WHEN the message broker initializes or topic metadata is queried
• THEN topic cdr.raw exists with 3 partitions, replication factor of 1, retention time of 86,400,000 ms (24 hours), and LZ4 compression enabled

Requirement: Dead Letter Queue Topic Configuration

The message broker SHALL provide a topic named cdr.raw.dlq to retain unprocessable or malformed records for inspection and replay.

Scenario: DLQ topic configuration applied

• WHEN the message broker initializes or topic metadata is queried
• THEN topic cdr.raw.dlq exists with 1 partition, replication factor of 1, and retention time of 604,800,000 ms (7 days)

Requirement: Standardized DLQ Message Envelope

All messages published to cdr.raw.dlq SHALL be wrapped in a standardized JSON envelope containing error diagnostic metadata.

Scenario: DLQ envelope structure enforced

• WHEN a message is routed to cdr.raw.dlq due to processing or validation failure
• THEN the message payload contains a JSON object with top-level fields: original_payload (object or string), failure_reason (string), failed_field (nullable string), failed_at (ISO 8601 UTC timestamp), and worker_instance (string identifier)

Scenario: Failure reason taxonomy enforced

• WHEN any record is routed to cdr.raw.dlq
• THEN the failure_reason value matches one of: missing_required_field, invalid_rat_value, invalid_epoch, malformed_json, or normalization_error
• THEN the failure_reason value matches one of: missing_required_field, invalid_rat_value, invalid_epoch, malformed_json, normalization_error