Files
tabdeal-job-queue/backend

Backend

This backend is a minimal production-aware PostgreSQL job queue implemented with Django and Django REST Framework.

The project intentionally avoids building a general queue platform. There is no Queue model, no Worker model, no Execution model, no Redis abstraction, and no public worker claim endpoint. The important interview requirement is implemented directly:

  • deterministic job state transitions
  • valid database state enforced by constraints
  • safe concurrent claiming with PostgreSQL row locks
  • retries with exponential backoff
  • worker leases and expired-lease recovery
  • append-only execution events for UI/demo visibility
  • environment-configured multi-threaded workers

The backend has only two queue-domain tables:

  • jobs_job
  • jobs_jobevent

All queue behavior is implemented in service functions under jobs/services.py.


High-Level Architecture

prompt for generating an svg image for the backend high-level architecture of a minimal Django REST Framework job queue: PostgreSQL at the center storing jobs_job and jobs_jobevent, Django REST API for create/list/retry/stats, separate Django worker command using the same service functions, N worker threads configured by environment variables, and demo handlers; use clean labeled boxes and arrows, no extra services like Redis or queue table

PostgreSQL
  |
  | stores valid queue state and append-only events
  |
Django REST API
  |
  | creates jobs, lists jobs, lists events, retries failed jobs, exposes stats
  |
Worker command
  |
  | imports the same backend service code
  | runs N worker threads configured by environment variables
  |
Demo handlers

The API process and worker process use the same codebase. In Docker Compose they are separate services using the same image:

backend -> python manage.py migrate && python manage.py runserver 0.0.0.0:8000
worker  -> python manage.py run_job_workers

The worker does not call HTTP endpoints to claim jobs. It imports claim_next_job, complete_job, fail_job, renew_lease, and cleanup_expired_jobs from jobs/services.py.


Design Scope

Included:

  • PostgreSQL-backed internal queue
  • one logical internal queue
  • priority scheduling
  • delayed eligibility through available_at
  • idempotency keys for job creation
  • deterministic retries with exponential backoff
  • worker leases
  • lease renewal for long-running handlers
  • expired lease cleanup
  • graceful worker shutdown
  • DRF API for jobs/events/stats
  • cursor pagination for global events
  • limit-offset pagination for jobs
  • Django Unfold admin

Explicitly not included:

  • queue CRUD
  • queue database table
  • worker database table
  • execution database table
  • cancellation state
  • batch claiming
  • Redis implementation
  • cron service
  • WebSockets/PostgreSQL LISTEN/NOTIFY
  • exactly-once execution guarantee

This is an at-least-once queue. A worker can perform a side effect and then crash before marking the job as succeeded. For real external work, handlers must be idempotent.


Core Queue Semantics

Statuses

prompt for generating an svg image for a deterministic job queue state machine with states queued, running, succeeded, failed; show valid transitions queued to running on claim, running to succeeded on completion, running to queued on retry or lease timeout, running to failed when attempts are exhausted, failed to queued on manual retry; include notes that invalid transitions are rejected by service ownership checks and database constraints

Job.status is intentionally small:

queued
running
succeeded
failed

Supported transitions:

queued  -> running
running -> succeeded
running -> queued      retry after handler failure or lease expiry
running -> failed      attempts exhausted
failed  -> queued      manual retry

Unsupported transitions are rejected by service-level ownership checks and by database shape constraints.

Ownership

prompt for generating an svg image for job ownership and stale worker protection in a PostgreSQL job queue: Worker A claims attempt 1 with locked_by and locked_until, lease expires, cleanup requeues, Worker B claims attempt 2, Worker A tries to complete attempt 1 and the database update affects zero rows because attempts and locked_by no longer match; use worker lanes, database row snapshots, and a clear rejected stale completion marker

A worker owns a job only when all of these are true:

status = running
locked_by = worker_id
attempts = attempt_number_that_worker_claimed
locked_until is in the future

Completion, failure, progress, and lease renewal all verify the worker still owns the same attempt. This prevents a stale worker from overwriting the result of a later attempt.

Example stale worker scenario:

1. Worker A claims job attempt 1.
2. Worker A hangs.
3. Lease expires.
4. Cleanup requeues the job.
5. Worker B claims attempt 2.
6. Worker A wakes up and tries to mark attempt 1 succeeded.
7. The update affects zero rows because attempts no longer matches.

This is one of the most important correctness properties in the project.


Database Model

prompt for generating an svg image for an ERD of a minimal two-table job queue database with jobs_job and jobs_jobevent; jobs_job has uuid id, type, payload, status, priority, available_at, attempts, max_attempts, idempotency_key, locked_by, locked_until, result, timestamps; jobs_jobevent has id, job_id foreign key, type, attempt, worker_id, message, data, created_at; show one-to-many relationship from jobs_job to jobs_jobevent and highlight that there is no queue table or worker table

Job

Job is the source of truth for queue state.

Important fields:

Field Purpose
id UUID primary key.
type Handler routing key, for example demo.success.
payload JSON input for the handler.
status One of queued, running, succeeded, failed.
priority Higher priority is claimed first. Default is 50.
available_at Earliest time the job can be claimed. Used for delayed jobs and retry backoff.
attempts Number of claim attempts already made.
max_attempts Maximum allowed attempts before final failure.
idempotency_key Optional unique key to prevent duplicate job creation.
locked_by Ephemeral worker identity for the current running attempt.
locked_until Lease expiry time for the current running attempt.
last_error Latest failure reason.
result JSON result for succeeded jobs.
finished_at Required for terminal statuses.

JobEvent

JobEvent is an append-only audit/event timeline for demo visibility and debugging.

Events are not the source of truth for queue state. The Job row is the source of truth. Events explain how the job reached its current state.

Supported event types:

created
claimed
progress
lease_renewed
succeeded
retry_scheduled
failed
timeout_requeued
manual_retry

Database Constraints

The model enforces valid job shape in the database.

Valid Status Shape

queued jobs:

locked_by is null
locked_until is null
finished_at is null

running jobs:

locked_by is not null
locked_until is not null
finished_at is null

succeeded and failed jobs:

locked_by is null
locked_until is null
finished_at is not null

Attempts Constraint

attempts >= 0
max_attempts >= 1
attempts <= max_attempts

Idempotency Constraint

idempotency_key is unique only when present:

unique idempotency_key where idempotency_key is not null

This allows many jobs with no idempotency key while preventing duplicate client-created jobs when a key is supplied.


Indexes

prompt for generating an svg image for PostgreSQL indexes used by the job queue: a claim path using partial index job_claim_idx on queued jobs ordered by available_at, priority descending, created_at, id; a timeout cleanup path using job_timeout_idx on running locked_until; and event lookup indexes for job timeline and global cursor feed; use three grouped panels with query arrows

Claim Index

job_claim_idx
fields: available_at, -priority, created_at, id
condition: status = queued

This supports the hot claim query:

WHERE status = queued
  AND available_at <= now()
  AND attempts < max_attempts
ORDER BY priority DESC, available_at ASC, created_at ASC, id ASC

Timeout Index

job_timeout_idx
fields: locked_until
condition: status = running

This supports expired lease cleanup.

Event Indexes

job_event_job_id_idx: job, id
job_event_id_idx: id

These support job detail timelines and global cursor-paginated event feeds.


ERD DBML

Copy this DBML into dbdiagram.io to generate the ERD.

Project minimal_job_queue {
  database_type: "PostgreSQL"
  Note: '''
    Minimal PostgreSQL-backed job queue.

    The queue domain intentionally has two tables:
    - jobs_job: current queue state and lock ownership
    - jobs_jobevent: append-only audit/event timeline

    There is no queue table, worker table, execution table, or task table.
  '''
}

Enum job_status {
  queued
  running
  succeeded
  failed
}

Enum job_event_type {
  created
  claimed
  progress
  lease_renewed
  succeeded
  retry_scheduled
  failed
  timeout_requeued
  manual_retry
}

Table jobs_job {
  id uuid [pk, not null, note: "UUID primary key generated by Django uuid.uuid4"]

  type varchar(120) [not null, note: "Handler routing key, for example demo.success"]
  payload jsonb [not null, default: `'{}'::jsonb`, note: "Handler input payload"]

  status job_status [not null, default: "queued", note: "queued | running | succeeded | failed"]
  priority smallint [not null, default: 50, note: "Higher priority is claimed first"]
  available_at timestamptz [not null, note: "Earliest time the job can be claimed"]

  attempts integer [not null, default: 0, note: "Number of attempts already claimed"]
  max_attempts integer [not null, default: 3, note: "Maximum attempts before final failure"]

  idempotency_key varchar(255) [unique, note: "Nullable unique key. Real DB uses a partial unique index where not null."]

  locked_by varchar(255) [note: "Ephemeral worker id for current running attempt"]
  locked_until timestamptz [note: "Lease expiry for current running attempt"]

  last_error text [not null, default: "", note: "Latest handler or lease error"]
  result jsonb [note: "JSON result after success"]

  created_at timestamptz [not null, note: "Created timestamp"]
  updated_at timestamptz [not null, note: "Updated timestamp"]
  finished_at timestamptz [note: "Set only for succeeded/failed jobs"]

  indexes {
    status [name: "jobs_job_status_idx"]
    available_at [name: "jobs_job_available_at_idx"]
    created_at [name: "jobs_job_created_at_idx"]
    (available_at, priority, created_at, id) [name: "job_claim_idx", note: "Partial index in PostgreSQL: WHERE status = 'queued'. priority is DESC in Django migration."]
    locked_until [name: "job_timeout_idx", note: "Partial index in PostgreSQL: WHERE status = 'running'."]
    idempotency_key [name: "job_idempotency_key_unique", unique, note: "Partial unique index in PostgreSQL: WHERE idempotency_key IS NOT NULL."]
  }

  Note: '''
    Check constraints implemented in Django/PostgreSQL:

    job_status_valid_shape:
      queued:
        locked_by IS NULL
        locked_until IS NULL
        finished_at IS NULL

      running:
        locked_by IS NOT NULL
        locked_until IS NOT NULL
        finished_at IS NULL

      succeeded/failed:
        locked_by IS NULL
        locked_until IS NULL
        finished_at IS NOT NULL

    job_attempts_valid:
      attempts >= 0
      max_attempts >= 1
      attempts <= max_attempts
  '''
}

Table jobs_jobevent {
  id bigint [pk, increment, not null]
  job_id uuid [not null, ref: > jobs_job.id]

  type job_event_type [not null, note: "created | claimed | progress | lease_renewed | succeeded | retry_scheduled | failed | timeout_requeued | manual_retry"]
  attempt integer [not null, default: 0]
  worker_id varchar(255) [note: "Worker identity that emitted the event, if applicable"]
  message text [not null, default: ""]
  data jsonb [not null, default: `'{}'::jsonb`]
  created_at timestamptz [not null]

  indexes {
    type [name: "jobs_jobevent_type_idx"]
    created_at [name: "jobs_jobevent_created_at_idx"]
    (job_id, id) [name: "job_event_job_id_idx"]
    id [name: "job_event_id_idx"]
  }

  Note: '''
    Append-only audit timeline.
    Job state is not derived from this table.
    The jobs_job row remains the source of truth.
  '''
}

Claiming Algorithm

prompt for generating an svg image for atomic job claiming with SELECT FOR UPDATE SKIP LOCKED: multiple worker threads query queued available jobs, one locks and updates a row to running, other workers skip locked rows and claim different jobs, then a claimed event is inserted; show a short transaction boundary around selection and update, and show handler execution outside the transaction

The worker claims a job in a short transaction:

queryset = Job.objects.filter(
    status=Job.Status.QUEUED,
    available_at__lte=now,
    attempts__lt=F("max_attempts"),
).order_by("-priority", "available_at", "created_at", "id")

queryset = queryset.select_for_update(skip_locked=True)
job = queryset.first()

Then the selected job is updated:

status       = running
attempts     = attempts + 1
locked_by    = worker_id
locked_until = now + lease_seconds
last_error   = ""
result       = null

And a claimed event is inserted.

Why SKIP LOCKED matters:

  • multiple worker threads can query at the same time
  • one worker locks a candidate row
  • other workers skip locked rows instead of blocking
  • each job attempt is claimed by at most one worker

The handler runs outside the claim transaction. The database row is not held locked while work is executing.


Completion Algorithm

Success is persisted only if the worker still owns the same attempt:

WHERE id = job_id
  AND status = running
  AND locked_by = worker_id
  AND attempts = attempt

If the update affects zero rows, the worker lost ownership. The service returns None and does not force the job to succeeded.

On success:

status       = succeeded
result       = handler result
locked_by    = null
locked_until = null
finished_at  = now

Then a succeeded event is inserted.


Failure And Retry Algorithm

prompt for generating an svg image for automatic exponential backoff retry in a job queue: running job fails, if attempts remain it returns to queued with available_at delayed by base * 2^(attempt-1), emits retry_scheduled, then later is claimed again; if max attempts is reached it transitions to failed; include a small backoff ladder 5s, 10s, 20s up to max 300s

On handler failure, the worker calls fail_job.

If attempts remain:

running -> queued
available_at = now + backoff_delay
locked_by = null
locked_until = null
last_error = error message
event = retry_scheduled

If attempts are exhausted:

running -> failed
locked_by = null
locked_until = null
finished_at = now
last_error = error message
event = failed

Backoff is deterministic:

delay_seconds = min(base_seconds * 2 ^ (attempt - 1), max_backoff_seconds)

With defaults:

attempt 1 -> 5 seconds
attempt 2 -> 10 seconds
attempt 3 -> 20 seconds
...
max delay -> 300 seconds

Lease Renewal

prompt for generating an svg image for lease renewal in long-running jobs: worker owns a running job with locked_until, handler periodically emits progress and renews the lease, locked_until moves forward only when worker_id and attempt match; show a contrasting timeout demo job that does not renew and becomes eligible for cleanup

Long-running handlers can renew their lease:

context.renew()

Lease renewal succeeds only if:

id = job_id
status = running
locked_by = worker_id
attempts = attempt

This means a stale worker cannot renew an old attempt after the job has already been requeued and claimed by another worker.

The demo demo.slow handler renews leases while it sleeps.

The demo demo.timeout handler intentionally does not renew leases so the UI can demonstrate lease expiry and requeue/failure behavior.


Expired Lease Cleanup

prompt for generating an svg image for expired lease cleanup: worker cleanup loop scans running jobs where locked_until is in the past, locks expired rows with SKIP LOCKED, requeues them with retry backoff when attempts remain or marks failed when attempts are exhausted, and emits timeout_requeued or failed events; use a flowchart with two decision branches

Every worker thread periodically runs cleanup:

status = running
locked_until < now

Expired jobs are locked with select_for_update(skip_locked=True).

If attempts remain:

running -> queued
available_at = now + backoff_delay
event = timeout_requeued

If attempts are exhausted:

running -> failed
event = failed

There is no separate cron process. This is intentional for the interview version: the worker process owns both execution and lease cleanup.


Worker Runtime

prompt for generating an svg image for the worker runtime loop: start worker runner, spawn JOB_WORKER_THREADS threads, each thread periodically cleanup expired leases, claim next job, sleep if none, execute handler, renew lease/progress during work, then complete or retry/fail, and stop claiming new jobs on SIGINT or SIGTERM; use a loop diagram with clear labels

Run workers with:

python manage.py run_job_workers

In Docker:

docker compose up worker

The worker runner:

  1. starts JOB_WORKER_THREADS Python threads
  2. gives each thread its own generated worker id
  3. periodically cleans expired leases
  4. claims one eligible job
  5. executes the matching handler
  6. marks the job succeeded, retry scheduled, or failed
  7. sleeps when no job is available
  8. stops claiming new jobs on SIGINT/SIGTERM

Worker id format:

hostname:pid:thread-number:short-random-id

Example:

638b98e627f5:1:thread-1:4e3b5893

Environment Variables

Django/PostgreSQL

Variable Default Purpose
DEBUG true Enables debug mode and permissive CORS in local development.
DJANGO_SECRET_KEY local dev key Django secret key. Override outside local development.
DJANGO_ALLOWED_HOSTS localhost,127.0.0.1,0.0.0.0 Allowed hosts.
DJANGO_CORS_ALLOWED_ORIGINS http://localhost:5173,http://127.0.0.1:5173 Frontend origins.
POSTGRES_DB job_queue Database name.
POSTGRES_USER postgres Database user.
POSTGRES_PASSWORD postgres Database password.
POSTGRES_HOST localhost Database host. In Docker this is postgres.
POSTGRES_PORT 5432 Database port.
POSTGRES_CONN_MAX_AGE 0 Django persistent DB connection age.
TIME_ZONE Asia/Tehran Django timezone.

Worker

Variable Default Purpose
JOB_WORKER_THREADS 4 Number of worker threads in the worker process.
JOB_WORKER_POLL_INTERVAL_MS 500 Sleep interval when no job is claimable.
JOB_WORKER_LEASE_SECONDS 30 Lock lease duration for a running attempt.
JOB_WORKER_CLEANUP_INTERVAL_SECONDS 5 How often each worker thread checks expired leases.
JOB_WORKER_CLEANUP_BATCH_SIZE 100 Max expired leases cleaned in one cleanup pass.
JOB_WORKER_BACKOFF_BASE_SECONDS 5 Base retry backoff.
JOB_WORKER_BACKOFF_MAX_SECONDS 300 Maximum retry delay.
JOB_WORKER_MAX_ATTEMPTS_DEFAULT 3 Default max attempts for jobs that do not specify one.
JOB_WORKER_LOG_LEVEL INFO Worker terminal log level. Use DEBUG for detailed polling/claim/execution logs.

API

prompt for generating an svg image for the backend API surface of the job queue: health endpoint, jobs create/list/detail/retry/stats endpoints, job events endpoint, global cursor-paginated event feed, schema/docs endpoints; group endpoints by health, jobs, events, documentation; use REST method badges and arrows to jobs_job/jobs_jobevent tables

Base URL in local development:

http://localhost:8000/api

Health

GET /api/health/

Response:

{
  "ok": true,
  "database": "ok"
}

If the database cannot be reached:

{
  "ok": false,
  "database": "error",
  "detail": "..."
}

Create Job

POST /api/jobs/

Body:

{
  "type": "demo.success",
  "payload": {
    "sleep_seconds": 1
  },
  "priority": 50,
  "available_at": null,
  "max_attempts": 3,
  "idempotency_key": "optional-client-key"
}

Behavior:

  • creates a queued job
  • sets available_at to now when omitted/null
  • emits a created event
  • returns 201 Created for a new job
  • returns 200 OK with the existing job when idempotency_key already exists

List Jobs

GET /api/jobs/?limit=25&offset=0&status=queued&type=demo

Pagination:

  • limit-offset pagination
  • default limit: 25
  • max limit: 100

Filters:

Query param Behavior
status Exact status match.
type Case-insensitive substring match against job type.

Response:

{
  "count": 53,
  "next": "http://localhost:8000/api/jobs/?limit=25&offset=25",
  "previous": null,
  "results": [
    {
      "id": "8421aca2-7d0c-484c-9c19-e7e324a39e21",
      "type": "demo.success",
      "payload": {
        "sleep_seconds": 1
      },
      "status": "queued",
      "priority": 50,
      "available_at": "2026-06-21T02:33:01.725170+03:30",
      "attempts": 0,
      "max_attempts": 3,
      "idempotency_key": null,
      "locked_by": null,
      "locked_until": null,
      "last_error": "",
      "result": null,
      "created_at": "2026-06-21T02:32:46.176967+03:30",
      "updated_at": "2026-06-21T02:32:46.176967+03:30",
      "finished_at": null
    }
  ]
}

Get Job Detail

GET /api/jobs/{id}/

Returns the same job shape as the list result item.

Get Events For One Job

GET /api/jobs/{id}/events/

Returns events ordered by ascending id:

[
  {
    "id": 1,
    "job": "8421aca2-7d0c-484c-9c19-e7e324a39e21",
    "type": "created",
    "attempt": 0,
    "worker_id": null,
    "message": "Job created",
    "data": {
      "type": "demo.success"
    },
    "created_at": "2026-06-21T02:32:46.176967+03:30"
  }
]

Global Event Feed

GET /api/job-events/?limit=50&type=failed&job_id={uuid}

Pagination:

  • cursor pagination
  • ordering: newest first by id
  • default page size: 50
  • max page size: 200

Filters:

Query param Behavior
type Exact event type match.
job_id Events for one job.

Response:

{
  "next": "http://localhost:8000/api/job-events/?cursor=cD05NzM%3D&limit=50",
  "previous": null,
  "results": [
    {
      "id": 973,
      "job": "8421aca2-7d0c-484c-9c19-e7e324a39e21",
      "type": "failed",
      "attempt": 3,
      "worker_id": "638b98e627f5:1:thread-1:4e3b5893",
      "message": "Intentional demo failure",
      "data": {
        "error": "Intentional demo failure"
      },
      "created_at": "2026-06-21T02:33:20.600665+03:30"
    }
  ]
}

Cursor pagination is used for the event feed because events are append-only and live. Offset pagination can drift when new rows are inserted while the user is paging.

Job Stats

GET /api/jobs/stats/

Response:

{
  "total": 53,
  "by_status": {
    "queued": 0,
    "running": 0,
    "succeeded": 41,
    "failed": 12
  },
  "overdue_running": 0,
  "retries_pending": 0,
  "oldest_queued_at": null
}

Retry Failed Job

POST /api/jobs/{id}/retry/

Only failed jobs can be manually retried.

Behavior:

failed -> queued
attempts = 0
available_at = now
lock fields cleared
last_error cleared
result cleared
finished_at cleared
event = manual_retry

Built-In Demo Job Types

Handlers are registered in jobs/handlers.py.

Type Behavior
demo.success Sleeps briefly, renews lease, emits progress, succeeds.
demo.fail Emits progress and raises an intentional error.
demo.slow Sleeps longer, renews lease, emits progress, succeeds.
demo.timeout Sleeps without lease renewal to demonstrate expired lease cleanup.
demo.flaky Fails until a configured attempt, then succeeds.

Example payloads:

{
  "sleep_seconds": 3
}
{
  "error": "Intentional demo failure"
}
{
  "fail_until_attempt": 2
}

Django Admin

The project uses django-unfold for the Django admin UI.

URL:

http://localhost:8000/admin/

Create an admin user:

docker compose exec backend python manage.py createsuperuser

The admin is useful for inspecting raw job state, event rows, locks, retry timestamps, and failed jobs.


Local Development

Install dependencies in the backend environment:

cd backend
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

Run migrations:

.\.venv\Scripts\python.exe manage.py migrate

Run API:

.\.venv\Scripts\python.exe manage.py runserver 0.0.0.0:8000

Run workers:

.\.venv\Scripts\python.exe manage.py run_job_workers

Run the full Docker stack from the repository root:

docker compose up --build

Rebuild only backend and worker:

docker compose up -d --build backend worker

Stop all Docker services:

docker compose down

Testing

Run backend checks:

cd backend
.\.venv\Scripts\python.exe -m ruff check .
$env:TEST_DATABASE_ENGINE='sqlite'; .\.venv\Scripts\python.exe -m pytest

The test suite covers:

  • successful claim
  • concurrent claim prevention on PostgreSQL
  • deterministic priority and available_at ordering
  • idempotent create
  • retry scheduling with exponential backoff
  • exhausted attempts becoming failed
  • expired lease cleanup
  • stale worker cannot complete old attempt
  • lease renewal ownership checks
  • manual retry
  • worker shutdown flag
  • jobs limit-offset pagination
  • global event cursor pagination

The PostgreSQL-specific concurrent claim test is skipped under SQLite because SKIP LOCKED is PostgreSQL behavior.


Assumptions And Simplifications

This backend intentionally chooses a small, defensible design for the interview task. The following are explicit assumptions and simplifications:

  • Single internal queue: there is no Queue table because queue routing is not part of the assignment. Job selection is controlled by priority, available_at, and deterministic ordering.
  • Ephemeral workers: there is no Worker table because workers self-identify through generated locked_by values and JobEvent.worker_id. Persisting worker rows would require registration, heartbeat cleanup, and stale-row management.
  • No execution table: JobEvent is enough for demo visibility and debugging. A larger production system may keep immutable per-attempt execution rows for duration metrics, artifacts, and richer attempt history.
  • No cancellation state: cancellation is omitted to keep the state machine small and focused on queued, running, succeeded, and failed.
  • No batch claiming: workers claim one job at a time. This keeps correctness and tests easier to reason about, although batch claiming can improve throughput.
  • No WebSockets or PostgreSQL LISTEN/NOTIFY: the UI uses polling and cursor pagination for deterministic demo behavior.
  • No authentication: the API is local/demo focused. Production would require authentication, authorization, and role-based controls for write/operator actions.
  • No retention policy: JobEvent grows forever in this version. Production should archive, partition, or delete old events.
  • Clock assumption: lease expiry depends on database and application clocks being reasonably synchronized.
  • Handler idempotency assumption: external side effects must tolerate retries because delivery is at least once.

Engineering Concepts Demonstrated

The implementation is small, but it deliberately demonstrates production-relevant backend concepts:

  • transactional state machine
  • row-level locking
  • SELECT ... FOR UPDATE SKIP LOCKED
  • partial indexes for hot queue queries
  • database check constraints for valid row shape
  • idempotency key for duplicate create protection
  • ownership guard using locked_by + attempts
  • lease-based failure recovery
  • automatic exponential backoff
  • at-least-once delivery semantics
  • cursor pagination for append-only event feeds
  • limit-offset pagination for bounded job lists
  • append-only audit log
  • graceful worker shutdown
  • environment-based runtime configuration

What Would Change For A Larger Production System

The current design is intentionally minimal. A larger production queue would likely add:

  • Redis or a dedicated broker for very high throughput and broader distributed workloads.
  • Authentication and authorization for all write endpoints and operator actions.
  • Cancellation or cooperative stop states for jobs that should no longer run.
  • Dead-letter metadata or a dead-letter inspection view for permanently failed jobs.
  • Metrics, alerting, and dashboards for queue depth, job age, throughput, failures, and retries.
  • Event retention, archival, or PostgreSQL partitioning for JobEvent.
  • A dedicated execution table if detailed per-attempt history is required.
  • Queue pause, drain, and resume controls for operational maintenance.
  • Batch claiming if single-job claims become a throughput bottleneck.

Operational Notes

At-Least-Once Delivery

This queue provides at-least-once execution.

It prevents two active workers from owning the same job attempt. It does not guarantee that a handler's external side effects happen exactly once.

Example:

1. Worker performs an external API call.
2. Worker process crashes before marking the job succeeded.
3. Lease expires.
4. Another worker retries the job.

The external API call may happen twice unless the handler is idempotent.

Why PostgreSQL

PostgreSQL is used because the assignment focuses on deterministic transactional state and safe concurrent claiming.

SELECT ... FOR UPDATE SKIP LOCKED gives a simple and reliable concurrency primitive for a small internal queue:

  • transactional claim
  • row-level ownership
  • no separate broker
  • easy inspection through SQL/admin/UI

For a high-throughput distributed production queue, Redis-backed tools such as BullMQ or Sidekiq are often better. This project intentionally stays PostgreSQL-first for interview clarity and fewer moving parts.

Why No Queue Model

There is one internal logical queue. type, priority, and available_at are enough for this demo.

Adding queue CRUD would add API surface and database complexity without improving the core correctness demonstration.

Why No Worker Model

Workers are ephemeral. A worker id is generated at process/thread startup and stored only in locked_by and events.

A worker table would require registration, heartbeat cleanup, stale row management, and admin decisions that are not needed for this assignment.

Why No Execution Model

Job.attempts stores the current attempt count, and JobEvent stores the audit timeline.

That is enough for this interview implementation. A separate immutable JobExecution table would be useful if the system needed detailed duration metrics, per-attempt artifacts, or long-term attempt history beyond events.


Important Files

File Purpose
config/settings.py Django, database, DRF, Unfold, and worker settings.
jobs/models.py Job and JobEvent models, constraints, indexes.
jobs/services.py Queue state machine and transactional service functions.
jobs/handlers.py Demo job handlers.
jobs/worker.py Threaded worker runner.
jobs/management/commands/run_job_workers.py Worker management command and logging setup.
jobs/views.py DRF API views.
jobs/serializers.py API serializers.
jobs/admin.py Django Unfold admin registration.
jobs/tests/ Backend tests.

Current API Summary

GET  /api/health/
GET  /api/config/
POST /api/jobs/
GET  /api/jobs/?limit=25&offset=0&status=queued&type=demo
GET  /api/jobs/{id}/
GET  /api/jobs/{id}/events/
GET  /api/job-events/?limit=50&type=failed&job_id={uuid}
GET  /api/jobs/stats/
POST /api/jobs/{id}/retry/
GET  /api/schema/
GET  /api/docs/