feat: add worker retry and replay safety
This commit is contained in:
13
README.md
13
README.md
@@ -81,6 +81,11 @@ worker processing is not connected yet.
|
||||
* Flush on 1,000 records or 2-second timeout
|
||||
* Kafka offsets committed only after successful ClickHouse batch writes
|
||||
* Remaining worker buffer flushed during graceful shutdown
|
||||
* ClickHouse insert retries with exponential backoff
|
||||
* Failed ClickHouse batches routed to the DLQ after three attempts
|
||||
* Whole-batch offset commits with no partial Kafka acknowledgement
|
||||
* Replay-safe storage using deterministic UUIDv5 identifiers and `ReplacingMergeTree`
|
||||
* Graceful worker shutdown flushes buffered records before exit
|
||||
|
||||
## Project Structure
|
||||
|
||||
@@ -205,3 +210,11 @@ pytest -v
|
||||
```
|
||||
|
||||
The current test suite covers domain validation, normalization, deterministic ID generation, and API request validation.
|
||||
|
||||
## Delivery Semantics
|
||||
|
||||
The worker uses at-least-once delivery.
|
||||
|
||||
Kafka offsets are committed only after a ClickHouse batch is successfully written or, after exhausted retries, the complete failed batch is acknowledged by the DLQ.
|
||||
|
||||
Repeated records use deterministic UUIDv5 identifiers and `ReplacingMergeTree` to support replay-safe storage.
|
||||
|
||||
Reference in New Issue
Block a user