docs(pipeline): align delivery contracts and verification requirements
This commit is contained in:
@@ -95,3 +95,25 @@ The Processing Worker SHALL handle `SIGTERM` and `SIGINT` signals to flush buffe
|
||||
#### Scenario: SIGTERM received during operation
|
||||
- **WHEN** a `SIGTERM` signal is received by the worker process
|
||||
- **THEN** the worker stops consuming new messages, flushes any remaining in-memory buffered records to ClickHouse, commits the associated Kafka offsets, and exits cleanly with exit code 0
|
||||
|
||||
### Requirement: Safe partition progress and confirmed DLQ
|
||||
The worker SHALL disable automatic offset storage, commit explicit next offsets, and await all destination acknowledgements.
|
||||
|
||||
#### Scenario: Valid record precedes an invalid record
|
||||
- **WHEN** an invalid record follows buffered valid records
|
||||
- **THEN** flush and commit those valid records before acknowledging and committing the invalid record to DLQ
|
||||
|
||||
#### Scenario: Destination or commit failure
|
||||
- **WHEN** DLQ delivery or any partition commit fails
|
||||
- **THEN** stop nonzero without advancing unsafe offsets or reclassifying the error as a ClickHouse failure; fatal cleanup does not flush the buffer
|
||||
|
||||
#### Scenario: Exhausted ClickHouse retries
|
||||
- **WHEN** three configured insert attempts fail
|
||||
- **THEN** publish original vendor payloads with ch_insert_failed and commit only after all DLQ deliveries succeed
|
||||
|
||||
### Requirement: Storage-safe normalization
|
||||
The worker SHALL reject numeric booleans, fractional numeric fields, out-of-range UInt64 and UInt32 values including total bytes, invalid IPv4, and timestamps outside DateTime range without crashing.
|
||||
|
||||
#### Scenario: Invalid scalar or conversion
|
||||
- **WHEN** any field fails its type, range or conversion check
|
||||
- **THEN** route only that record to DLQ with a field-specific reason and preserve other valid records
|
||||
|
||||
Reference in New Issue
Block a user