38 lines
2.5 KiB
Markdown
38 lines
2.5 KiB
Markdown
## Purpose
|
|
|
|
Defines the ClickHouse database schema, table engines, deduplication semantics, and automated idempotent initialization for storing normalized CDRs.
|
|
|
|
## ADDED Requirements
|
|
|
|
### Requirement: Database and Table Schema Definition
|
|
The storage layer SHALL provide a database named `telecom` and a table named `cdr_records` optimized for high-volume append and analytical queries.
|
|
|
|
#### Scenario: Database and table initialized idempotently
|
|
- **WHEN** the initialization script or container entrypoint executes
|
|
- **THEN** ClickHouse creates database `telecom` (`CREATE DATABASE IF NOT EXISTS`) and table `telecom.cdr_records` (`CREATE TABLE IF NOT EXISTS`) using `ReplacingMergeTree(processed_at)`, partitioned by `toYYYYMM(session_start_time)`, ordered by `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`, with `TTL session_start_time + INTERVAL 90 DAY` and `SETTINGS index_granularity = 8192`
|
|
|
|
### Requirement: Deduplication on At-Least-Once Replay
|
|
The storage layer SHALL support idempotent re-insertion of records without creating duplicate entries in analytical views.
|
|
|
|
#### Scenario: Duplicate record inserted after crash replay
|
|
- **WHEN** a batch containing previously written records is re-inserted due to consumer restart before offset commit
|
|
- **THEN** ClickHouse accepts the rows into `ReplacingMergeTree(processed_at)` and deduplicates rows sharing the same order key upon merge or when queried with `FINAL`
|
|
|
|
### Requirement: Source IP Column Type Compatibility
|
|
The storage layer and worker client SHALL store `source_ip` using the ClickHouse `IPv4` data type.
|
|
|
|
#### Scenario: IPv4 string inserted into table
|
|
- **WHEN** a normalized record with an IPv4 address string (e.g. `"10.240.12.1"`) is inserted
|
|
- **THEN** the value is inserted as a valid IPv4 representation compatible with ClickHouse's internal `IPv4` (`UInt32`) structure using `toIPv4()` casting or native client IP types
|
|
|
|
### Requirement: Identity preservation and schema compatibility
|
|
Initialization SHALL use one canonical schema, verify existing table compatibility and never reset data automatically.
|
|
|
|
#### Scenario: Distinct identities share analytical metadata
|
|
- **WHEN** records have different raw_id values but identical other sorting fields
|
|
- **THEN** both survive FINAL and background replacement, while exact replay yields one logical row
|
|
|
|
#### Scenario: Existing incompatible table
|
|
- **WHEN** an existing table lacks the corrected identity sorting key
|
|
- **THEN** initialization fails with explicit assessment reset instructions without deleting data
|