tabdeal-job-queue/README.md

Job Queue

This is a deliberately small PostgreSQL-backed job queue for the interview assignment.

The important parts are:

• one internal priority queue
• two tables: jobs and job_events
• atomic claiming with PostgreSQL row locks and SKIP LOCKED
• deterministic status transitions
• automatic retries with exponential backoff
• lease renewal for long-running jobs
• lease timeout cleanup when workers die
• idempotent job creation
• demo UI with job state and event polling

Architecture

There is no queue table and no worker table. Workers are ephemeral process threads with generated ids. The queue is internal and ordered by:

priority DESC, available_at ASC, created_at ASC, id ASC

Statuses

The database also validates row shape:

• queued jobs cannot have locks or finish timestamps
• running jobs must have a lock owner and lease deadline
• terminal jobs must have a finish timestamp and no lock

At-Least-Once Execution

This queue provides at-least-once execution, not exactly-once execution.

A worker can perform an external side effect and crash before marking a job succeeded. The lease will expire and the job can run again. Real handlers should therefore be idempotent.

Why PostgreSQL

The assignment requires PostgreSQL, and PostgreSQL gives a compact solution for safe concurrent claiming through SELECT ... FOR UPDATE SKIP LOCKED. This keeps the implementation transactional, inspectable, and easy to demo.

For a high-throughput distributed production queue, Redis-backed systems such as BullMQ or Sidekiq-style designs are common. That is documented as the next architecture, not implemented here.

Assumptions And Interview Simplifications

This project is intentionally scoped as an internal single-queue system, not a multi-tenant queue platform. There is no queue CRUD, queue table, or dynamic routing model because the assignment focuses on safe claim semantics and deterministic job state.

PostgreSQL is used because the assignment requires it and because it makes transactional state easy to inspect during a demo. Redis, BullMQ, Sidekiq-style designs, or a dedicated broker would be better for very high throughput or broader distributed queue use cases.

The system provides at-least-once execution, not exactly-once execution. Handlers that perform external side effects must be idempotent because a worker can crash after the side effect and before marking the job succeeded.

The UI is demo and observability oriented. It shows queue pressure, job state, retries, leases, and event logs, but it is not a full production operator console.

Authentication and authorization are intentionally omitted from the public API for interview simplicity. A production deployment would protect all write endpoints and usually restrict operator actions by role.

Production Follow-Ups

• Add authentication, authorization, and role-based permissions.
• Add metrics, alerting, and dashboards for queue depth, age, throughput, failures, and retry rate.
• Add event retention, archival, or partitioning so JobEvent does not grow forever.
• Add rate limits, producer quotas, and backpressure controls.
• Add cancellation or cooperative stop support for jobs.
• Add dead-letter metadata or a dead-letter inspection view.
• Consider Redis or a dedicated broker if throughput or cross-service distribution becomes the main requirement.

Run

copy .env.sample .env
npm.cmd --prefix frontend install
npm.cmd --prefix frontend run build
docker compose up --build

Open:

http://localhost:5173

API docs:

http://localhost:8000/api/docs/

Admin panel:

http://localhost:8000/admin/

Create an admin user:

docker compose exec backend python manage.py createsuperuser

Worker logs default to INFO. To see every poll, claim, lease renewal, progress update, retry, and completion, set:

JOB_WORKER_LOG_LEVEL=DEBUG

Then recreate the worker:

docker compose up -d --build worker
docker compose logs -f worker

Useful Commands

Run backend tests locally with SQLite fallback:

$env:TEST_DATABASE_ENGINE="sqlite"
python manage.py test

Run pytest:

$env:TEST_DATABASE_ENGINE="sqlite"
python -m pytest

Run worker locally:

python manage.py run_job_workers

References

• PostgreSQL SKIP LOCKED: https://www.postgresql.org/docs/current/sql-select.html
• pg-boss: https://github.com/timgit/pg-boss
• Solid Queue: https://github.com/rails/solid_queue
• BullMQ concurrency: https://docs.bullmq.io/guide/workers/concurrency
• Distributed task queue article: https://medium.com/@sindhukripa007/i-built-a-distributed-task-queue-from-scratch-to-actually-understand-how-they-work-37fa0452ff9b