telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md

Purpose

Defines the ClickHouse database schema, table engines, deduplication semantics, and automated idempotent initialization for storing normalized CDRs.

ADDED Requirements

Requirement: Database and Table Schema Definition

The storage layer SHALL provide a database named telecom and a table named cdr_records optimized for high-volume append and analytical queries.

Scenario: Database and table initialized idempotently

• WHEN the initialization script or container entrypoint executes
• THEN ClickHouse creates database telecom (CREATE DATABASE IF NOT EXISTS) and table telecom.cdr_records (CREATE TABLE IF NOT EXISTS) using ReplacingMergeTree(processed_at), partitioned by toYYYYMM(session_start_time), ordered by (service_type, radio_access_type, cell_id, session_start_time, msisdn), with TTL session_start_time + INTERVAL 90 DAY and SETTINGS index_granularity = 8192

Requirement: Deduplication on At-Least-Once Replay

The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.

Scenario: Duplicate record inserted after crash replay

• WHEN a batch containing previously written records is re-inserted due to consumer restart before offset commit
• THEN ClickHouse accepts the rows into ReplacingMergeTree(processed_at) and deduplicates rows sharing the same order key upon merge or when queried with FINAL

Requirement: Source IP Column Type Compatibility

The storage layer and worker client SHALL store source_ip using the ClickHouse IPv4 data type.

Scenario: IPv4 string inserted into table

• WHEN a normalized record with an IPv4 address string (e.g. "10.240.12.1") is inserted
• THEN the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal IPv4 (UInt32) structure using toIPv4() casting or native client IP types