docs: clarify queue assumptions and tradeoffs
This commit is contained in:
@@ -1032,6 +1032,61 @@ The PostgreSQL-specific concurrent claim test is skipped under SQLite because `S
|
||||
|
||||
---
|
||||
|
||||
## Assumptions And Simplifications
|
||||
|
||||
This backend intentionally chooses a small, defensible design for the interview task. The following are explicit assumptions and simplifications:
|
||||
|
||||
- **Single internal queue**: there is no `Queue` table because queue routing is not part of the assignment. Job selection is controlled by `priority`, `available_at`, and deterministic ordering.
|
||||
- **Ephemeral workers**: there is no `Worker` table because workers self-identify through generated `locked_by` values and `JobEvent.worker_id`. Persisting worker rows would require registration, heartbeat cleanup, and stale-row management.
|
||||
- **No execution table**: `JobEvent` is enough for demo visibility and debugging. A larger production system may keep immutable per-attempt execution rows for duration metrics, artifacts, and richer attempt history.
|
||||
- **No cancellation state**: cancellation is omitted to keep the state machine small and focused on `queued`, `running`, `succeeded`, and `failed`.
|
||||
- **No batch claiming**: workers claim one job at a time. This keeps correctness and tests easier to reason about, although batch claiming can improve throughput.
|
||||
- **No WebSockets or PostgreSQL `LISTEN/NOTIFY`**: the UI uses polling and cursor pagination for deterministic demo behavior.
|
||||
- **No authentication**: the API is local/demo focused. Production would require authentication, authorization, and role-based controls for write/operator actions.
|
||||
- **No retention policy**: `JobEvent` grows forever in this version. Production should archive, partition, or delete old events.
|
||||
- **Clock assumption**: lease expiry depends on database and application clocks being reasonably synchronized.
|
||||
- **Handler idempotency assumption**: external side effects must tolerate retries because delivery is at least once.
|
||||
|
||||
---
|
||||
|
||||
## Engineering Concepts Demonstrated
|
||||
|
||||
The implementation is small, but it deliberately demonstrates production-relevant backend concepts:
|
||||
|
||||
- transactional state machine
|
||||
- row-level locking
|
||||
- `SELECT ... FOR UPDATE SKIP LOCKED`
|
||||
- partial indexes for hot queue queries
|
||||
- database check constraints for valid row shape
|
||||
- idempotency key for duplicate create protection
|
||||
- ownership guard using `locked_by + attempts`
|
||||
- lease-based failure recovery
|
||||
- automatic exponential backoff
|
||||
- at-least-once delivery semantics
|
||||
- cursor pagination for append-only event feeds
|
||||
- limit-offset pagination for bounded job lists
|
||||
- append-only audit log
|
||||
- graceful worker shutdown
|
||||
- environment-based runtime configuration
|
||||
|
||||
---
|
||||
|
||||
## What Would Change For A Larger Production System
|
||||
|
||||
The current design is intentionally minimal. A larger production queue would likely add:
|
||||
|
||||
- Redis or a dedicated broker for very high throughput and broader distributed workloads.
|
||||
- Authentication and authorization for all write endpoints and operator actions.
|
||||
- Cancellation or cooperative stop states for jobs that should no longer run.
|
||||
- Dead-letter metadata or a dead-letter inspection view for permanently failed jobs.
|
||||
- Metrics, alerting, and dashboards for queue depth, job age, throughput, failures, and retries.
|
||||
- Event retention, archival, or PostgreSQL partitioning for `JobEvent`.
|
||||
- A dedicated execution table if detailed per-attempt history is required.
|
||||
- Queue pause, drain, and resume controls for operational maintenance.
|
||||
- Batch claiming if single-job claims become a throughput bottleneck.
|
||||
|
||||
---
|
||||
|
||||
## Operational Notes
|
||||
|
||||
### At-Least-Once Delivery
|
||||
|
||||
Reference in New Issue
Block a user