138 lines
3.3 KiB
Markdown
138 lines
3.3 KiB
Markdown
# Minimal Senior-Level Job Queue
|
|
|
|
This is a deliberately small PostgreSQL-backed job queue for the interview assignment.
|
|
|
|
The important parts are:
|
|
|
|
- one internal priority queue
|
|
- two tables: `jobs` and `job_events`
|
|
- atomic claiming with PostgreSQL row locks and `SKIP LOCKED`
|
|
- deterministic status transitions
|
|
- automatic retries with exponential backoff
|
|
- lease renewal for long-running jobs
|
|
- lease timeout cleanup when workers die
|
|
- idempotent job creation
|
|
- demo UI with job state and event polling
|
|
|
|
## Run
|
|
|
|
```powershell
|
|
copy .env.sample .env
|
|
npm.cmd --prefix frontend install
|
|
npm.cmd --prefix frontend run build
|
|
docker compose up --build
|
|
```
|
|
|
|
Open:
|
|
|
|
```text
|
|
http://localhost:5173
|
|
```
|
|
|
|
API docs:
|
|
|
|
```text
|
|
http://localhost:8000/api/docs/
|
|
```
|
|
|
|
Admin panel:
|
|
|
|
```text
|
|
http://localhost:8000/admin/
|
|
```
|
|
|
|
Create an admin user:
|
|
|
|
```powershell
|
|
docker compose exec backend python manage.py createsuperuser
|
|
```
|
|
|
|
Worker logs default to `INFO`. To see every poll, claim, lease renewal, progress update, retry, and completion, set:
|
|
|
|
```env
|
|
JOB_WORKER_LOG_LEVEL=DEBUG
|
|
```
|
|
|
|
Then recreate the worker:
|
|
|
|
```powershell
|
|
docker compose up -d --build worker
|
|
docker compose logs -f worker
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
React UI
|
|
|
|
|
Django API ---- PostgreSQL
|
|
|
|
|
Django worker process
|
|
|
|
|
N worker threads from env
|
|
```
|
|
|
|
There is no queue table and no worker table. Workers are ephemeral process threads with generated ids. The queue is internal and ordered by:
|
|
|
|
```text
|
|
priority DESC, available_at ASC, created_at ASC, id ASC
|
|
```
|
|
|
|
## Statuses
|
|
|
|
```text
|
|
queued -> running
|
|
running -> succeeded
|
|
running -> queued retry after failure or timeout
|
|
running -> failed attempts exhausted
|
|
failed -> queued manual retry
|
|
```
|
|
|
|
The database also validates row shape:
|
|
|
|
- queued jobs cannot have locks or finish timestamps
|
|
- running jobs must have a lock owner and lease deadline
|
|
- terminal jobs must have a finish timestamp and no lock
|
|
|
|
## At-Least-Once Execution
|
|
|
|
This queue provides at-least-once execution, not exactly-once execution.
|
|
|
|
A worker can perform an external side effect and crash before marking a job succeeded. The lease will expire and the job can run again. Real handlers should therefore be idempotent.
|
|
|
|
## Why PostgreSQL
|
|
|
|
The assignment requires PostgreSQL, and PostgreSQL gives a compact solution for safe concurrent claiming through `SELECT ... FOR UPDATE SKIP LOCKED`. This keeps the implementation transactional, inspectable, and easy to demo.
|
|
|
|
For a high-throughput distributed production queue, Redis-backed systems such as BullMQ or Sidekiq-style designs are common. That is documented as the next architecture, not implemented here.
|
|
|
|
## Useful Commands
|
|
|
|
Run backend tests locally with SQLite fallback:
|
|
|
|
```powershell
|
|
$env:TEST_DATABASE_ENGINE="sqlite"
|
|
python manage.py test
|
|
```
|
|
|
|
Run pytest:
|
|
|
|
```powershell
|
|
$env:TEST_DATABASE_ENGINE="sqlite"
|
|
python -m pytest
|
|
```
|
|
|
|
Run worker locally:
|
|
|
|
```powershell
|
|
python manage.py run_job_workers
|
|
```
|
|
|
|
## References
|
|
|
|
- PostgreSQL `SKIP LOCKED`: https://www.postgresql.org/docs/current/sql-select.html
|
|
- pg-boss: https://github.com/timgit/pg-boss
|
|
- Solid Queue: https://github.com/rails/solid_queue
|
|
- BullMQ concurrency: https://docs.bullmq.io/guide/workers/concurrency
|
|
- Distributed task queue article: https://medium.com/@sindhukripa007/i-built-a-distributed-task-queue-from-scratch-to-actually-understand-how-they-work-37fa0452ff9b
|