telecom-cdr-pipeline-ai/openspec/changes/telecom-cdr-pipeline/specs/clickhouse-storage/spec.md

Purpose

Defines the ClickHouse database schema, table engines, deduplication semantics, and automated idempotent initialization for storing normalized CDRs.

ADDED Requirements

Requirement: Database and Table Schema Definition

The storage layer SHALL provide a database named telecom and a table named cdr_records optimized for high-volume append and analytical queries.

Scenario: Database and table initialized idempotently

• WHEN the initialization script or container entrypoint executes
• THEN ClickHouse creates database telecom (CREATE DATABASE IF NOT EXISTS) and table telecom.cdr_records (CREATE TABLE IF NOT EXISTS) using ReplacingMergeTree(processed_at), partitioned by toYYYYMM(session_start_time), ordered by (service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id), with TTL session_start_time + INTERVAL 90 DAY and SETTINGS index_granularity = 8192

Requirement: Deduplication on At-Least-Once Replay

The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.

Scenario: Duplicate record inserted after crash replay

• WHEN a batch containing previously written records is re-inserted due to consumer restart before offset commit
• THEN ClickHouse accepts the rows into ReplacingMergeTree(processed_at) and deduplicates exact replays sharing the deterministic cdr_id-bearing order key upon merge or when queried with FINAL, without collapsing distinct raw_id values whose session metadata happens to match

Requirement: Source IP Column Type Compatibility

The storage layer and worker client SHALL store source_ip using the ClickHouse IPv4 data type.

Scenario: IPv4 string inserted into table

• WHEN a normalized record with an IPv4 address string (e.g. "10.240.12.1") is inserted
• THEN the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal IPv4 (UInt32) structure using toIPv4() casting or native client IP types

Requirement: Identity preservation and schema compatibility

Initialization SHALL use one canonical schema, verify existing table compatibility and never reset data automatically.

Scenario: Distinct identities share analytical metadata

• WHEN records have different raw_id values but identical other sorting fields
• THEN both survive FINAL and background replacement, while an exact replay yields one logical row

Scenario: Existing incompatible table

• WHEN an existing table does not use the canonical identity-bearing sorting key
• THEN initialization fails with explicit assessment reset instructions without deleting data