feat: add worker retry and replay safety

This commit is contained in:
2026-09-20 15:59:46 +03:30
parent 1a2f492992
commit 1021fe55b4
7 changed files with 188 additions and 23 deletions

View File

@@ -81,6 +81,11 @@ worker processing is not connected yet.
* Flush on 1,000 records or 2-second timeout
* Kafka offsets committed only after successful ClickHouse batch writes
* Remaining worker buffer flushed during graceful shutdown
* ClickHouse insert retries with exponential backoff
* Failed ClickHouse batches routed to the DLQ after three attempts
* Whole-batch offset commits with no partial Kafka acknowledgement
* Replay-safe storage using deterministic UUIDv5 identifiers and `ReplacingMergeTree`
* Graceful worker shutdown flushes buffered records before exit
## Project Structure
@@ -205,3 +210,11 @@ pytest -v
```
The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.
## Delivery Semantics
The worker uses at-least-once delivery.
Kafka offsets are committed only after a ClickHouse batch is successfully written or, after exhausted retries, the complete failed batch is acknowledged by the DLQ.
Repeated records use deterministic UUIDv5 identifiers and `ReplacingMergeTree` to support replay-safe storage.