docs(pipeline): align delivery contracts and verification requirements

This commit is contained in:
2026-09-19 21:31:05 +03:30
parent 977f838eb2
commit bf439ab03c
10 changed files with 209 additions and 53 deletions

View File

@@ -95,3 +95,25 @@ The Processing Worker SHALL handle `SIGTERM` and `SIGINT` signals to flush buffe
#### Scenario: SIGTERM received during operation
- **WHEN** a `SIGTERM` signal is received by the worker process
- **THEN** the worker stops consuming new messages, flushes any remaining in-memory buffered records to ClickHouse, commits the associated Kafka offsets, and exits cleanly with exit code 0
### Requirement: Safe partition progress and confirmed DLQ
The worker SHALL disable automatic offset storage, commit explicit next offsets, and await all destination acknowledgements.
#### Scenario: Valid record precedes an invalid record
- **WHEN** an invalid record follows buffered valid records
- **THEN** flush and commit those valid records before acknowledging and committing the invalid record to DLQ
#### Scenario: Destination or commit failure
- **WHEN** DLQ delivery or any partition commit fails
- **THEN** stop nonzero without advancing unsafe offsets or reclassifying the error as a ClickHouse failure; fatal cleanup does not flush the buffer
#### Scenario: Exhausted ClickHouse retries
- **WHEN** three configured insert attempts fail
- **THEN** publish original vendor payloads with ch_insert_failed and commit only after all DLQ deliveries succeed
### Requirement: Storage-safe normalization
The worker SHALL reject numeric booleans, fractional numeric fields, out-of-range UInt64 and UInt32 values including total bytes, invalid IPv4, and timestamps outside DateTime range without crashing.
#### Scenario: Invalid scalar or conversion
- **WHEN** any field fails its type, range or conversion check
- **THEN** route only that record to DLQ with a field-specific reason and preserve other valid records