docs(specs): initialize OpenSpec and define telecom CDR pipeline specifications
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
## Purpose
|
||||
|
||||
Defines the ClickHouse database schema, table engines, deduplication semantics, and automated idempotent initialization for storing normalized CDRs.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Database and Table Schema Definition
|
||||
The storage layer SHALL provide a database named `telecom` and a table named `cdr_records` optimized for high-volume append and analytical queries.
|
||||
|
||||
#### Scenario: Database and table initialized idempotently
|
||||
- **WHEN** the initialization script or container entrypoint executes
|
||||
- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
|
||||
|
||||
### Requirement: Deduplication on At-Least-Once Replay
|
||||
The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.
|
||||
|
||||
#### Scenario: Duplicate record inserted after crash replay
|
||||
- **WHEN** a batch containing previously written records is re-inserted due to consumer restart before offset commit
|
||||
- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates rows sharing the same order key upon merge or when queried with `FINAL`
|
||||
|
||||
### Requirement: Source IP Column Type Compatibility
|
||||
The storage layer and worker client SHALL store `source_ip` using the ClickHouse `IPv4` data type.
|
||||
|
||||
#### Scenario: IPv4 string inserted into table
|
||||
- **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
|
||||
- **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types
|
||||
|
||||
Reference in New Issue
Block a user