diff --git a/README.md b/README.md new file mode 100644 index 0000000..313af8e --- /dev/null +++ b/README.md @@ -0,0 +1,112 @@ +# Minimal Senior-Level Job Queue + +This is a deliberately small PostgreSQL-backed job queue for the interview assignment. + +The important parts are: + +- one internal priority queue +- two tables: `jobs` and `job_events` +- atomic claiming with PostgreSQL row locks and `SKIP LOCKED` +- deterministic status transitions +- automatic retries with exponential backoff +- lease renewal for long-running jobs +- lease timeout cleanup when workers die +- idempotent job creation +- demo UI with job state and event polling + +## Run + +```powershell +copy .env.sample .env +npm.cmd --prefix frontend install +npm.cmd --prefix frontend run build +docker compose up --build +``` + +Open: + +```text +http://localhost:5173 +``` + +API docs: + +```text +http://localhost:8000/api/docs/ +``` + +## Architecture + +```text +React UI + | +Django API ---- PostgreSQL + | +Django worker process + | +N worker threads from env +``` + +There is no queue table and no worker table. Workers are ephemeral process threads with generated ids. The queue is internal and ordered by: + +```text +priority DESC, available_at ASC, created_at ASC, id ASC +``` + +## Statuses + +```text +queued -> running +running -> succeeded +running -> queued retry after failure or timeout +running -> failed attempts exhausted +failed -> queued manual retry +``` + +The database also validates row shape: + +- queued jobs cannot have locks or finish timestamps +- running jobs must have a lock owner and lease deadline +- terminal jobs must have a finish timestamp and no lock + +## At-Least-Once Execution + +This queue provides at-least-once execution, not exactly-once execution. + +A worker can perform an external side effect and crash before marking a job succeeded. The lease will expire and the job can run again. Real handlers should therefore be idempotent. + +## Why PostgreSQL + +The assignment requires PostgreSQL, and PostgreSQL gives a compact solution for safe concurrent claiming through `SELECT ... FOR UPDATE SKIP LOCKED`. This keeps the implementation transactional, inspectable, and easy to demo. + +For a high-throughput distributed production queue, Redis-backed systems such as BullMQ or Sidekiq-style designs are common. That is documented as the next architecture, not implemented here. + +## Useful Commands + +Run backend tests locally with SQLite fallback: + +```powershell +$env:TEST_DATABASE_ENGINE="sqlite" +python manage.py test +``` + +Run pytest: + +```powershell +$env:TEST_DATABASE_ENGINE="sqlite" +python -m pytest +``` + +Run worker locally: + +```powershell +python manage.py run_job_workers +``` + +## References + +- PostgreSQL `SKIP LOCKED`: https://www.postgresql.org/docs/current/sql-select.html +- pg-boss: https://github.com/timgit/pg-boss +- Solid Queue: https://github.com/rails/solid_queue +- BullMQ concurrency: https://docs.bullmq.io/guide/workers/concurrency +- Distributed task queue article: https://medium.com/@sindhukripa007/i-built-a-distributed-task-queue-from-scratch-to-actually-understand-how-they-work-37fa0452ff9b