85 lines
9.3 KiB
Markdown
85 lines
9.3 KiB
Markdown
## Context
|
|
|
|
See `proposal.md` for background and problem motivation.
|
|
|
|
The architecture is constrained to pure Python without web frameworks (FastAPI, Flask, Starlette, or Django). The pipeline must ingest high-throughput telecom CDRs, decouple ingestion from processing via Kafka, normalize heterogeneous fields, micro-batch into ClickHouse with at-least-once delivery guarantees, and provide complete operational observability via Prometheus metrics and structured JSON logging.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Implement a non-blocking HTTP ingestion server using Python standard library `asyncio` primitives.
|
|
- Enforce backpressure using bounded async queues (`429 Too Many Requests`).
|
|
- Publish to Kafka topic `cdr.raw` using `raw_id` partition keys with `confluent-kafka`.
|
|
- Build a resilient worker that normalizes phone numbers (E.164 via `phonenumbers`), maps RATs, generates deterministic UUIDv5 `cdr_id`s, and safely routes bad data to `cdr.raw.dlq`.
|
|
- Micro-batch inserts into ClickHouse (`ReplacingMergeTree`) with configurable count (1,000) and time (2s) flush thresholds.
|
|
- Guarantee at-least-once delivery by committing Kafka offsets only after ClickHouse write ACK.
|
|
- Expose dual-service Prometheus `/metrics` and aggregate public `/stats` via an internal worker HTTP endpoint.
|
|
- Emit structured JSON logs across all services.
|
|
- Containerize all services in `docker-compose.yml` with health checks and `condition: service_healthy`.
|
|
|
|
**Non-Goals:**
|
|
- Web-based visualization UI (analytical queries and verification scripts provide reporting).
|
|
- Schema Registry broker integration (deferred from optional bonus challenges).
|
|
- Distributed multi-worker consumer coordination (single worker instance fulfills baseline requirements).
|
|
|
|
## Decisions
|
|
|
|
### 1. Framework-less `asyncio` HTTP Server
|
|
- **Choice**: Implement the HTTP listener using `asyncio.start_server` with a lightweight, zero-dependency HTTP/1.1 request parser.
|
|
- **Rationale**: Strictly satisfies the constraint of "pure Python without frameworks" while delivering high concurrent connection handling and a bounded set of asynchronous delivery futures for backpressure.
|
|
- **Alternatives Considered**:
|
|
- `http.server.ThreadingHTTPServer`: Standard library, but thread-per-request model has higher memory overhead and makes bounded async backpressure coordination harder.
|
|
- `FastAPI`/`Starlette`: Disallowed by project constraints.
|
|
|
|
### 2. Service Separation and Stats Aggregation
|
|
- **Choice**: Run `Ingestion API` and `Processing Worker` as separate Docker containers. The public `/stats` endpoint is hosted by the Ingestion API, which queries a lightweight internal HTTP endpoint on the Worker (`GET http://worker:8001/internal/stats`) and aggregates with local ingestion metrics.
|
|
- **Rationale**: Isolates HTTP ingestion from processing CPU/network bottlenecks. Enables independent scaling and lifecycle management.
|
|
- **Alternatives Considered**:
|
|
- Single combined monolithic process: Violates explicit requirement for separate services.
|
|
- Querying ClickHouse directly for `/stats`: Incurs database query overhead and does not report real-time in-flight buffer size or consumer lag.
|
|
|
|
### 3. Kafka Client and Partitioning Strategy
|
|
- **Choice**: Use `confluent-kafka` with `raw_id` as the message key when publishing to `cdr.raw`.
|
|
- **Rationale**: `confluent-kafka` is built on `librdkafka` and offers superior throughput over pure Python clients. Using `raw_id` ensures deterministic partition affinity for duplicate CDR submissions while spreading traffic evenly across the 3 partitions using the configured Kafka client partitioner.
|
|
- **Alternatives Considered**:
|
|
- `caller_num` as key: Risk of partition hotspots if certain subscribers generate disproportionate traffic.
|
|
- `null` (round-robin): Loses record-level partition affinity.
|
|
|
|
### 4. ClickHouse Client and Idempotent Deduplication
|
|
- **Choice**: Use `clickhouse-connect` HTTP client targeting table `telecom.cdr_records` with engine `ReplacingMergeTree(processed_at)`.
|
|
- **Rationale**: ClickHouse recommends HTTP batch inserts. `ReplacingMergeTree` combined with deterministic UUIDv5 `cdr_id` ensures that if a batch is replayed after a worker crash prior to offset commit, duplicate records with identical order keys are safely merged without data duplication.
|
|
- **Alternatives Considered**:
|
|
- `clickhouse-driver` (native TCP protocol): Slightly higher client-side setup complexity; `clickhouse-connect` is the official modern ClickHouse client.
|
|
- Standard `MergeTree`: Does not deduplicate replayed batches, violating idempotency.
|
|
|
|
### 5. Micro-Batching and Fault Tolerance Strategy
|
|
- **Choice**: Worker accumulates records into an in-memory batch. Flushes when `len(batch) >= BATCH_SIZE` (default 1,000) or elapsed time >= `BATCH_FLUSH_INTERVAL_SECONDS` (default 2s).
|
|
- **Failure Handling**:
|
|
- If ClickHouse write times out, worker retries with exponential backoff up to `CH_RETRY_MAX_ATTEMPTS` (default 3, initial delay 500ms, factor 2.0).
|
|
- If all retries fail, the entire batch is wrapped and routed to `cdr.raw.dlq`, and offsets are committed only after all DLQ acknowledgements.
|
|
- On `SIGTERM`, consumer stops, remaining buffer is flushed to ClickHouse, offsets are committed, and worker exits cleanly.
|
|
|
|
### 6. Structured JSON Logging
|
|
- **Choice**: Standard library Python logging with a custom JSON formatter producing fields: `timestamp`, `level`, `service`, `raw_id`, and `event`.
|
|
- **Rationale**: Clean, zero-dependency implementation producing machine-readable logs compatible with modern observability stacks.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **[Risk] High ingestion bursts saturate in-memory queue** → **[Mitigation]** The bounded outstanding-publication set rejects requests with `429 Too Many Requests` as soon as capacity is reached, protecting process memory.
|
|
- **[Risk] ClickHouse outage stalls worker** → **[Mitigation]** Retries are bounded to 3 attempts with exponential backoff. If ClickHouse remains unreachable, batch is offloaded to DLQ and offsets committed only after delivery acknowledgements; a delivery/commit failure stops consumption.
|
|
- **[Risk] Worker crashes between ClickHouse ACK and Kafka commit** → **[Mitigation]** Re-consumed batch is re-inserted; `ReplacingMergeTree(processed_at)` deduplicates rows sharing the order key `(service_type, radio_access_type, cell_id, session_start_time, msisdn, cdr_id)`.
|
|
- **[Risk] Phone number parsing performance overhead** → **[Mitigation]** Use `phonenumbers` library with fast-path prefix checks (`00`, `011`, `+`) before full parsing.
|
|
|
|
## Corrected delivery contracts
|
|
- API validates UTF-8, finite JSON, required non-null fields and exact types before admission. Semantic RAT, range, phone and IPv4 checks remain in the worker. Return structured 400 errors; 413 for bodies exceeding API_MAX_BODY_BYTES (65536), 408 for reads exceeding API_READ_TIMEOUT_SECONDS (10).
|
|
- Producer returns concurrent delivery futures and polls callbacks continuously. Queued plus outstanding publications are bounded by API_QUEUE_MAX_SIZE. Return 202 after acknowledgement, 429 at capacity, 503 on failure/timeout (KAFKA_DELIVERY_TIMEOUT_SECONDS=10). Timed-out delivery is ambiguous; retry identical raw_id and payload. Recover without a sticky health gate.
|
|
- Worker disables both automatic offset storage and commits. Before routing an invalid record, flush older buffered records. Await DLQ acknowledgements, then commit explicit next offsets per partition. Failed delivery/commit stops processing nonzero without advancing unsafe offsets; never classify commit failures as ClickHouse failures.
|
|
- ClickHouse retries at most three times with exponential delays between attempts. Exhausted batches retain original payloads in ch_insert_failed envelopes. Partial DLQ success permits duplicate replay, never missing source records.
|
|
- raw_id identifies immutable events; replay payloads must be identical. Include cdr_id in the replacement key, use UTC DateTimes and native IPv4 values, retain monthly partitioning and 90-day TTL. Refuse an incompatible existing schema; documented explicit reset only.
|
|
- Stop API admission and drain acknowledgements on SIGTERM/SIGINT. API and Linux worker shutdown budgets are 45 seconds; Compose grace is 60 seconds. Windows-host workers do not provide the POSIX alarm deadline. Worker fatal cleanup does not flush or commit unsafe state.
|
|
- Consumer lag is high watermark minus committed next offset (or low watermark before the first commit), refreshed periodically. Latency measures Kafka timestamp to acknowledged ClickHouse write. Counters reflect physical operations, FINAL rows reflect logical events. Escape labels and keep log message plus event.
|
|
- Compose defaults work without .env, while .env.example uses redpanda:29092. Ports, topics, credentials and schema names propagate to provisioning, clients and health checks. Provisioning verifies existing settings and fails on incompatible infrastructure.
|
|
- Verification sends 1570 HTTP requests (1540 accepted, 30 rejected), plus 30 direct Kafka injections; expect 1500 logical ClickHouse rows and 50 actual DLQ records. DLQ reads use run identifiers and a separate consumer group. The baseline is single-worker; live migration and conflicting identity updates are excluded.
|
|
|
|
Compose keeps the ClickHouse internal HTTP port at 8123; CLICKHOUSE_PORT selects the published host port. The verifier must fail when actual Kafka evidence is unavailable. Current follow-up edits have not been executed or validated by user instruction.
|