docs: clarify queue assumptions and tradeoffs

This commit is contained in:
2026-06-21 11:15:20 +03:30
parent bd358f5e48
commit b331948491
2 changed files with 77 additions and 0 deletions

View File

@@ -49,6 +49,28 @@ The assignment requires PostgreSQL, and PostgreSQL gives a compact solution for
For a high-throughput distributed production queue, Redis-backed systems such as BullMQ or Sidekiq-style designs are common. That is documented as the next architecture, not implemented here.
## Assumptions And Interview Simplifications
This project is intentionally scoped as an internal single-queue system, not a multi-tenant queue platform. There is no queue CRUD, queue table, or dynamic routing model because the assignment focuses on safe claim semantics and deterministic job state.
PostgreSQL is used because the assignment requires it and because it makes transactional state easy to inspect during a demo. Redis, BullMQ, Sidekiq-style designs, or a dedicated broker would be better for very high throughput or broader distributed queue use cases.
The system provides at-least-once execution, not exactly-once execution. Handlers that perform external side effects must be idempotent because a worker can crash after the side effect and before marking the job succeeded.
The UI is demo and observability oriented. It shows queue pressure, job state, retries, leases, and event logs, but it is not a full production operator console.
Authentication and authorization are intentionally omitted from the public API for interview simplicity. A production deployment would protect all write endpoints and usually restrict operator actions by role.
## Production Follow-Ups
- Add authentication, authorization, and role-based permissions.
- Add metrics, alerting, and dashboards for queue depth, age, throughput, failures, and retry rate.
- Add event retention, archival, or partitioning so `JobEvent` does not grow forever.
- Add rate limits, producer quotas, and backpressure controls.
- Add cancellation or cooperative stop support for jobs.
- Add dead-letter metadata or a dead-letter inspection view.
- Consider Redis or a dedicated broker if throughput or cross-service distribution becomes the main requirement.
## Run
```powershell

View File

@@ -1032,6 +1032,61 @@ The PostgreSQL-specific concurrent claim test is skipped under SQLite because `S
---
## Assumptions And Simplifications
This backend intentionally chooses a small, defensible design for the interview task. The following are explicit assumptions and simplifications:
- **Single internal queue**: there is no `Queue` table because queue routing is not part of the assignment. Job selection is controlled by `priority`, `available_at`, and deterministic ordering.
- **Ephemeral workers**: there is no `Worker` table because workers self-identify through generated `locked_by` values and `JobEvent.worker_id`. Persisting worker rows would require registration, heartbeat cleanup, and stale-row management.
- **No execution table**: `JobEvent` is enough for demo visibility and debugging. A larger production system may keep immutable per-attempt execution rows for duration metrics, artifacts, and richer attempt history.
- **No cancellation state**: cancellation is omitted to keep the state machine small and focused on `queued`, `running`, `succeeded`, and `failed`.
- **No batch claiming**: workers claim one job at a time. This keeps correctness and tests easier to reason about, although batch claiming can improve throughput.
- **No WebSockets or PostgreSQL `LISTEN/NOTIFY`**: the UI uses polling and cursor pagination for deterministic demo behavior.
- **No authentication**: the API is local/demo focused. Production would require authentication, authorization, and role-based controls for write/operator actions.
- **No retention policy**: `JobEvent` grows forever in this version. Production should archive, partition, or delete old events.
- **Clock assumption**: lease expiry depends on database and application clocks being reasonably synchronized.
- **Handler idempotency assumption**: external side effects must tolerate retries because delivery is at least once.
---
## Engineering Concepts Demonstrated
The implementation is small, but it deliberately demonstrates production-relevant backend concepts:
- transactional state machine
- row-level locking
- `SELECT ... FOR UPDATE SKIP LOCKED`
- partial indexes for hot queue queries
- database check constraints for valid row shape
- idempotency key for duplicate create protection
- ownership guard using `locked_by + attempts`
- lease-based failure recovery
- automatic exponential backoff
- at-least-once delivery semantics
- cursor pagination for append-only event feeds
- limit-offset pagination for bounded job lists
- append-only audit log
- graceful worker shutdown
- environment-based runtime configuration
---
## What Would Change For A Larger Production System
The current design is intentionally minimal. A larger production queue would likely add:
- Redis or a dedicated broker for very high throughput and broader distributed workloads.
- Authentication and authorization for all write endpoints and operator actions.
- Cancellation or cooperative stop states for jobs that should no longer run.
- Dead-letter metadata or a dead-letter inspection view for permanently failed jobs.
- Metrics, alerting, and dashboards for queue depth, job age, throughput, failures, and retries.
- Event retention, archival, or PostgreSQL partitioning for `JobEvent`.
- A dedicated execution table if detailed per-attempt history is required.
- Queue pause, drain, and resume controls for operational maintenance.
- Batch claiming if single-job claims become a throughput bottleneck.
---
## Operational Notes
### At-Least-Once Delivery