docs(specs): initialize OpenSpec and define telecom CDR pipeline specifications

This commit is contained in:
2026-09-19 13:09:00 +03:30
commit 5f75a1a2be
28 changed files with 2866 additions and 0 deletions

View File

@@ -0,0 +1,27 @@
## Purpose
Defines the ClickHouse database schema, table engines, deduplication semantics, and automated idempotent initialization for storing normalized CDRs.
## ADDED Requirements
### Requirement: Database and Table Schema Definition
The storage layer SHALL provide a database named `telecom` and a table named `cdr_records` optimized for high-volume append and analytical queries.
#### Scenario: Database and table initialized idempotently
- **WHEN** the initialization script or container entrypoint executes
- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
### Requirement: Deduplication on At-Least-Once Replay
The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.
#### Scenario: Duplicate record inserted after crash replay
- **WHEN** a batch containing previously written records is re-inserted due to consumer restart before offset commit
- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates rows sharing the same order key upon merge or when queried with `FINAL`
### Requirement: Source IP Column Type Compatibility
The storage layer and worker client SHALL store `source_ip` using the ClickHouse `IPv4` data type.
#### Scenario: IPv4 string inserted into table
- **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
- **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types