tabdeal-job-queue/backend/README.md

# Backend

This backend is a minimal production-aware PostgreSQL job queue implemented with Django and Django REST Framework.

The project intentionally avoids building a general queue platform. There is no `Queue` model, no `Worker` model, no `Execution` model, no Redis abstraction, and no public worker claim endpoint. The important interview requirement is implemented directly:

- deterministic job state transitions
- valid database state enforced by constraints
- safe concurrent claiming with PostgreSQL row locks
- retries with exponential backoff
- worker leases and expired-lease recovery
- append-only execution events for UI/demo visibility
- environment-configured multi-threaded workers

The backend has only two queue-domain tables:

- `jobs_job`
- `jobs_jobevent`

All queue behavior is implemented in service functions under `jobs/services.py`.

---

## High-Level Architecture

![prompt for generating an svg image for the backend high-level architecture of a minimal Django REST Framework job queue: PostgreSQL at the center storing jobs_job and jobs_jobevent, Django REST API for create/list/retry/stats, separate Django worker command using the same service functions, N worker threads configured by environment variables, and demo handlers; use clean labeled boxes and arrows, no extra services like Redis or queue table](../assets/images/backend/high-level-architecture.png)

```text
PostgreSQL
  |
  | stores valid queue state and append-only events
  |
Django REST API
  |
  | creates jobs, lists jobs, lists events, retries failed jobs, exposes stats
  |
Worker command
  |
  | imports the same backend service code
  | runs N worker threads configured by environment variables
  |
Demo handlers
```

The API process and worker process use the same codebase. In Docker Compose they are separate services using the same image:

```text
backend -> python manage.py migrate && python manage.py runserver 0.0.0.0:8000
worker  -> python manage.py run_job_workers
```

The worker does not call HTTP endpoints to claim jobs. It imports `claim_next_job`, `complete_job`, `fail_job`, `renew_lease`, and `cleanup_expired_jobs` from `jobs/services.py`.

---

## Design Scope

Included:

- PostgreSQL-backed internal queue
- one logical internal queue
- priority scheduling
- delayed eligibility through `available_at`
- idempotency keys for job creation
- deterministic retries with exponential backoff
- worker leases
- lease renewal for long-running handlers
- expired lease cleanup
- graceful worker shutdown
- DRF API for jobs/events/stats
- cursor pagination for global events
- limit-offset pagination for jobs
- Django Unfold admin

Explicitly not included:

- queue CRUD
- queue database table
- worker database table
- execution database table
- cancellation state
- batch claiming
- Redis implementation
- cron service
- WebSockets/PostgreSQL `LISTEN/NOTIFY`
- exactly-once execution guarantee

This is an at-least-once queue. A worker can perform a side effect and then crash before marking the job as succeeded. For real external work, handlers must be idempotent.

---

## Core Queue Semantics

### Statuses

![prompt for generating an svg image for a deterministic job queue state machine with states queued, running, succeeded, failed; show valid transitions queued to running on claim, running to succeeded on completion, running to queued on retry or lease timeout, running to failed when attempts are exhausted, failed to queued on manual retry; include notes that invalid transitions are rejected by service ownership checks and database constraints](../assets/images/backend/job-state-machine.png)

`Job.status` is intentionally small:

```text
queued
running
succeeded
failed
```

Supported transitions:

```text
queued  -> running
running -> succeeded
running -> queued      retry after handler failure or lease expiry
running -> failed      attempts exhausted
failed  -> queued      manual retry
```

Unsupported transitions are rejected by service-level ownership checks and by database shape constraints.

### Ownership

![prompt for generating an svg image for job ownership and stale worker protection in a PostgreSQL job queue: Worker A claims attempt 1 with locked_by and locked_until, lease expires, cleanup requeues, Worker B claims attempt 2, Worker A tries to complete attempt 1 and the database update affects zero rows because attempts and locked_by no longer match; use worker lanes, database row snapshots, and a clear rejected stale completion marker](../assets/images/backend/ownership-stale-worker.png)

A worker owns a job only when all of these are true:

```text
status = running
locked_by = worker_id
attempts = attempt_number_that_worker_claimed
locked_until is in the future
```

Completion, failure, progress, and lease renewal all verify the worker still owns the same attempt. This prevents a stale worker from overwriting the result of a later attempt.

Example stale worker scenario:

```text
1. Worker A claims job attempt 1.
2. Worker A hangs.
3. Lease expires.
4. Cleanup requeues the job.
5. Worker B claims attempt 2.
6. Worker A wakes up and tries to mark attempt 1 succeeded.
7. The update affects zero rows because attempts no longer matches.
```

This is one of the most important correctness properties in the project.

---

## Database Model

![prompt for generating an svg image for an ERD of a minimal two-table job queue database with jobs_job and jobs_jobevent; jobs_job has uuid id, type, payload, status, priority, available_at, attempts, max_attempts, idempotency_key, locked_by, locked_until, result, timestamps; jobs_jobevent has id, job_id foreign key, type, attempt, worker_id, message, data, created_at; show one-to-many relationship from jobs_job to jobs_jobevent and highlight that there is no queue table or worker table](../assets/images/backend/database-erd.png)

### `Job`

`Job` is the source of truth for queue state.

Important fields:

| Field | Purpose |
| --- | --- |
| `id` | UUID primary key. |
| `type` | Handler routing key, for example `demo.success`. |
| `payload` | JSON input for the handler. |
| `status` | One of `queued`, `running`, `succeeded`, `failed`. |
| `priority` | Higher priority is claimed first. Default is `50`. |
| `available_at` | Earliest time the job can be claimed. Used for delayed jobs and retry backoff. |
| `attempts` | Number of claim attempts already made. |
| `max_attempts` | Maximum allowed attempts before final failure. |
| `idempotency_key` | Optional unique key to prevent duplicate job creation. |
| `locked_by` | Ephemeral worker identity for the current running attempt. |
| `locked_until` | Lease expiry time for the current running attempt. |
| `last_error` | Latest failure reason. |
| `result` | JSON result for succeeded jobs. |
| `finished_at` | Required for terminal statuses. |

### `JobEvent`

`JobEvent` is an append-only audit/event timeline for demo visibility and debugging.

Events are not the source of truth for queue state. The `Job` row is the source of truth. Events explain how the job reached its current state.

Supported event types:

```text
created
claimed
progress
lease_renewed
succeeded
retry_scheduled
failed
timeout_requeued
manual_retry
```

---

## Database Constraints

The model enforces valid job shape in the database.

### Valid Status Shape

`queued` jobs:

```text
locked_by is null
locked_until is null
finished_at is null
```

`running` jobs:

```text
locked_by is not null
locked_until is not null
finished_at is null
```

`succeeded` and `failed` jobs:

```text
locked_by is null
locked_until is null
finished_at is not null
```

### Attempts Constraint

```text
attempts >= 0
max_attempts >= 1
attempts <= max_attempts
```

### Idempotency Constraint

`idempotency_key` is unique only when present:

```text
unique idempotency_key where idempotency_key is not null
```

This allows many jobs with no idempotency key while preventing duplicate client-created jobs when a key is supplied.

---

## Indexes

![prompt for generating an svg image for PostgreSQL indexes used by the job queue: a claim path using partial index job_claim_idx on queued jobs ordered by available_at, priority descending, created_at, id; a timeout cleanup path using job_timeout_idx on running locked_until; and event lookup indexes for job timeline and global cursor feed; use three grouped panels with query arrows](../assets/images/backend/database-indexes.png)

### Claim Index

```text
job_claim_idx
fields: available_at, -priority, created_at, id
condition: status = queued
```

This supports the hot claim query:

```text
WHERE status = queued
  AND available_at <= now()
  AND attempts < max_attempts
ORDER BY priority DESC, available_at ASC, created_at ASC, id ASC
```

### Timeout Index

```text
job_timeout_idx
fields: locked_until
condition: status = running
```

This supports expired lease cleanup.

### Event Indexes

```text
job_event_job_id_idx: job, id
job_event_id_idx: id
```

These support job detail timelines and global cursor-paginated event feeds.

---

## ERD DBML

Copy this DBML into [dbdiagram.io](https://dbdiagram.io) to generate the ERD.

```dbml
Project minimal_job_queue {
  database_type: "PostgreSQL"
  Note: '''
    Minimal PostgreSQL-backed job queue.

    The queue domain intentionally has two tables:
    - jobs_job: current queue state and lock ownership
    - jobs_jobevent: append-only audit/event timeline

    There is no queue table, worker table, execution table, or task table.
  '''
}

Enum job_status {
  queued
  running
  succeeded
  failed
}

Enum job_event_type {
  created
  claimed
  progress
  lease_renewed
  succeeded
  retry_scheduled
  failed
  timeout_requeued
  manual_retry
}

Table jobs_job {
  id uuid [pk, not null, note: "UUID primary key generated by Django uuid.uuid4"]

  type varchar(120) [not null, note: "Handler routing key, for example demo.success"]
  payload jsonb [not null, default: `'{}'::jsonb`, note: "Handler input payload"]

  status job_status [not null, default: "queued", note: "queued | running | succeeded | failed"]
  priority smallint [not null, default: 50, note: "Higher priority is claimed first"]
  available_at timestamptz [not null, note: "Earliest time the job can be claimed"]

  attempts integer [not null, default: 0, note: "Number of attempts already claimed"]
  max_attempts integer [not null, default: 3, note: "Maximum attempts before final failure"]

  idempotency_key varchar(255) [unique, note: "Nullable unique key. Real DB uses a partial unique index where not null."]

  locked_by varchar(255) [note: "Ephemeral worker id for current running attempt"]
  locked_until timestamptz [note: "Lease expiry for current running attempt"]

  last_error text [not null, default: "", note: "Latest handler or lease error"]
  result jsonb [note: "JSON result after success"]

  created_at timestamptz [not null, note: "Created timestamp"]
  updated_at timestamptz [not null, note: "Updated timestamp"]
  finished_at timestamptz [note: "Set only for succeeded/failed jobs"]

  indexes {
    status [name: "jobs_job_status_idx"]
    available_at [name: "jobs_job_available_at_idx"]
    created_at [name: "jobs_job_created_at_idx"]
    (available_at, priority, created_at, id) [name: "job_claim_idx", note: "Partial index in PostgreSQL: WHERE status = 'queued'. priority is DESC in Django migration."]
    locked_until [name: "job_timeout_idx", note: "Partial index in PostgreSQL: WHERE status = 'running'."]
    idempotency_key [name: "job_idempotency_key_unique", unique, note: "Partial unique index in PostgreSQL: WHERE idempotency_key IS NOT NULL."]
  }

  Note: '''
    Check constraints implemented in Django/PostgreSQL:

    job_status_valid_shape:
      queued:
        locked_by IS NULL
        locked_until IS NULL
        finished_at IS NULL

      running:
        locked_by IS NOT NULL
        locked_until IS NOT NULL
        finished_at IS NULL

      succeeded/failed:
        locked_by IS NULL
        locked_until IS NULL
        finished_at IS NOT NULL

    job_attempts_valid:
      attempts >= 0
      max_attempts >= 1
      attempts <= max_attempts
  '''
}

Table jobs_jobevent {
  id bigint [pk, increment, not null]
  job_id uuid [not null, ref: > jobs_job.id]

  type job_event_type [not null, note: "created | claimed | progress | lease_renewed | succeeded | retry_scheduled | failed | timeout_requeued | manual_retry"]
  attempt integer [not null, default: 0]
  worker_id varchar(255) [note: "Worker identity that emitted the event, if applicable"]
  message text [not null, default: ""]
  data jsonb [not null, default: `'{}'::jsonb`]
  created_at timestamptz [not null]

  indexes {
    type [name: "jobs_jobevent_type_idx"]
    created_at [name: "jobs_jobevent_created_at_idx"]
    (job_id, id) [name: "job_event_job_id_idx"]
    id [name: "job_event_id_idx"]
  }

  Note: '''
    Append-only audit timeline.
    Job state is not derived from this table.
    The jobs_job row remains the source of truth.
  '''
}
```

---

## Claiming Algorithm

![prompt for generating an svg image for atomic job claiming with SELECT FOR UPDATE SKIP LOCKED: multiple worker threads query queued available jobs, one locks and updates a row to running, other workers skip locked rows and claim different jobs, then a claimed event is inserted; show a short transaction boundary around selection and update, and show handler execution outside the transaction](../assets/images/backend/skip-locked-claim.png)

The worker claims a job in a short transaction:

```python
queryset = Job.objects.filter(
    status=Job.Status.QUEUED,
    available_at__lte=now,
    attempts__lt=F("max_attempts"),
).order_by("-priority", "available_at", "created_at", "id")

queryset = queryset.select_for_update(skip_locked=True)
job = queryset.first()
```

Then the selected job is updated:

```text
status       = running
attempts     = attempts + 1
locked_by    = worker_id
locked_until = now + lease_seconds
last_error   = ""
result       = null
```

And a `claimed` event is inserted.

Why `SKIP LOCKED` matters:

- multiple worker threads can query at the same time
- one worker locks a candidate row
- other workers skip locked rows instead of blocking
- each job attempt is claimed by at most one worker

The handler runs outside the claim transaction. The database row is not held locked while work is executing.

---

## Completion Algorithm

Success is persisted only if the worker still owns the same attempt:

```text
WHERE id = job_id
  AND status = running
  AND locked_by = worker_id
  AND attempts = attempt
```

If the update affects zero rows, the worker lost ownership. The service returns `None` and does not force the job to succeeded.

On success:

```text
status       = succeeded
result       = handler result
locked_by    = null
locked_until = null
finished_at  = now
```

Then a `succeeded` event is inserted.

---

## Failure And Retry Algorithm

![prompt for generating an svg image for automatic exponential backoff retry in a job queue: running job fails, if attempts remain it returns to queued with available_at delayed by base * 2^(attempt-1), emits retry_scheduled, then later is claimed again; if max attempts is reached it transitions to failed; include a small backoff ladder 5s, 10s, 20s up to max 300s](../assets/images/backend/exponential-backoff.png)

On handler failure, the worker calls `fail_job`.

If attempts remain:

```text
running -> queued
available_at = now + backoff_delay
locked_by = null
locked_until = null
last_error = error message
event = retry_scheduled
```

If attempts are exhausted:

```text
running -> failed
locked_by = null
locked_until = null
finished_at = now
last_error = error message
event = failed
```

Backoff is deterministic:

```text
delay_seconds = min(base_seconds * 2 ^ (attempt - 1), max_backoff_seconds)
```

With defaults:

```text
attempt 1 -> 5 seconds
attempt 2 -> 10 seconds
attempt 3 -> 20 seconds
...
max delay -> 300 seconds
```

---

## Lease Renewal

![prompt for generating an svg image for lease renewal in long-running jobs: worker owns a running job with locked_until, handler periodically emits progress and renews the lease, locked_until moves forward only when worker_id and attempt match; show a contrasting timeout demo job that does not renew and becomes eligible for cleanup](../assets/images/backend/lease-renewal.png)

Long-running handlers can renew their lease:

```python
context.renew()
```

Lease renewal succeeds only if:

```text
id = job_id
status = running
locked_by = worker_id
attempts = attempt
```

This means a stale worker cannot renew an old attempt after the job has already been requeued and claimed by another worker.

The demo `demo.slow` handler renews leases while it sleeps.

The demo `demo.timeout` handler intentionally does not renew leases so the UI can demonstrate lease expiry and requeue/failure behavior.

---

## Expired Lease Cleanup

![prompt for generating an svg image for expired lease cleanup: worker cleanup loop scans running jobs where locked_until is in the past, locks expired rows with SKIP LOCKED, requeues them with retry backoff when attempts remain or marks failed when attempts are exhausted, and emits timeout_requeued or failed events; use a flowchart with two decision branches](../assets/images/backend/expired-lease-cleanup.png)

Every worker thread periodically runs cleanup:

```text
status = running
locked_until < now
```

Expired jobs are locked with `select_for_update(skip_locked=True)`.

If attempts remain:

```text
running -> queued
available_at = now + backoff_delay
event = timeout_requeued
```

If attempts are exhausted:

```text
running -> failed
event = failed
```

There is no separate cron process. This is intentional for the interview version: the worker process owns both execution and lease cleanup.

---

## Worker Runtime

![prompt for generating an svg image for the worker runtime loop: start worker runner, spawn JOB_WORKER_THREADS threads, each thread periodically cleanup expired leases, claim next job, sleep if none, execute handler, renew lease/progress during work, then complete or retry/fail, and stop claiming new jobs on SIGINT or SIGTERM; use a loop diagram with clear labels](../assets/images/backend/worker-runtime-loop.png)

Run workers with:

```powershell
python manage.py run_job_workers
```

In Docker:

```powershell
docker compose up worker
```

The worker runner:

1. starts `JOB_WORKER_THREADS` Python threads
2. gives each thread its own generated worker id
3. periodically cleans expired leases
4. claims one eligible job
5. executes the matching handler
6. marks the job succeeded, retry scheduled, or failed
7. sleeps when no job is available
8. stops claiming new jobs on `SIGINT`/`SIGTERM`

Worker id format:

```text
hostname:pid:thread-number:short-random-id
```

Example:

```text
638b98e627f5:1:thread-1:4e3b5893
```

---

## Environment Variables

### Django/PostgreSQL

| Variable | Default | Purpose |
| --- | --- | --- |
| `DEBUG` | `true` | Enables debug mode and permissive CORS in local development. |
| `ENABLE_DJANGO_DEBUG_TOOLBAR` | `true` when `DEBUG=true` | Enables Django Debug Toolbar for local backend requests. |
| `DJANGO_INTERNAL_IPS` | `127.0.0.1,localhost,host.docker.internal` | Internal IP allowlist used by Django Debug Toolbar. |
| `DJANGO_SECRET_KEY` | local dev key | Django secret key. Override outside local development. |
| `DJANGO_ALLOWED_HOSTS` | `localhost,127.0.0.1,0.0.0.0` | Allowed hosts. |
| `DJANGO_CORS_ALLOWED_ORIGINS` | `http://localhost:5173,http://127.0.0.1:5173` | Frontend origins. |
| `POSTGRES_DB` | `job_queue` | Database name. |
| `POSTGRES_USER` | `postgres` | Database user. |
| `POSTGRES_PASSWORD` | `postgres` | Database password. |
| `POSTGRES_HOST` | `localhost` | Database host. In Docker this is `postgres`. |
| `POSTGRES_PORT` | `5432` | Database port. |
| `POSTGRES_CONN_MAX_AGE` | `0` | Django persistent DB connection age. |
| `TIME_ZONE` | `Asia/Tehran` | Django timezone. |

### Worker

| Variable | Default | Purpose |
| --- | --- | --- |
| `JOB_WORKER_THREADS` | `4` | Number of worker threads in the worker process. |
| `JOB_WORKER_POLL_INTERVAL_MS` | `500` | Sleep interval when no job is claimable. |
| `JOB_WORKER_LEASE_SECONDS` | `30` | Lock lease duration for a running attempt. |
| `JOB_WORKER_CLEANUP_INTERVAL_SECONDS` | `5` | How often each worker thread checks expired leases. |
| `JOB_WORKER_CLEANUP_BATCH_SIZE` | `100` | Max expired leases cleaned in one cleanup pass. |
| `JOB_WORKER_BACKOFF_BASE_SECONDS` | `5` | Base retry backoff. |
| `JOB_WORKER_BACKOFF_MAX_SECONDS` | `300` | Maximum retry delay. |
| `JOB_WORKER_MAX_ATTEMPTS_DEFAULT` | `3` | Default max attempts for jobs that do not specify one. |
| `JOB_WORKER_LOG_LEVEL` | `INFO` | Worker terminal log level. Use `DEBUG` for detailed polling/claim/execution logs. |

---

## API

![prompt for generating an svg image for the backend API surface of the job queue: health endpoint, jobs create/list/detail/retry/stats endpoints, job events endpoint, global cursor-paginated event feed, schema/docs endpoints; group endpoints by health, jobs, events, documentation; use REST method badges and arrows to jobs_job/jobs_jobevent tables](../assets/images/backend/api-surface.png)

Base URL in local development:

```text
http://localhost:8000/api
```

### Health

```http
GET /api/health/
```

Response:

```json
{
  "ok": true,
  "database": "ok"
}
```

If the database cannot be reached:

```json
{
  "ok": false,
  "database": "error",
  "detail": "..."
}
```

### Create Job

```http
POST /api/jobs/
```

Body:

```json
{
  "type": "demo.success",
  "payload": {
    "sleep_seconds": 1
  },
  "priority": 50,
  "available_at": null,
  "max_attempts": 3,
  "idempotency_key": "optional-client-key"
}
```

Behavior:

- creates a `queued` job
- sets `available_at` to now when omitted/null
- emits a `created` event
- returns `201 Created` for a new job
- returns `200 OK` with the existing job when `idempotency_key` already exists

### List Jobs

```http
GET /api/jobs/?limit=25&offset=0&status=queued&type=demo
```

Pagination:

- limit-offset pagination
- default limit: `25`
- max limit: `100`

Filters:

| Query param | Behavior |
| --- | --- |
| `status` | Exact status match. |
| `type` | Case-insensitive substring match against job type. |

Response:

```json
{
  "count": 53,
  "next": "http://localhost:8000/api/jobs/?limit=25&offset=25",
  "previous": null,
  "results": [
    {
      "id": "8421aca2-7d0c-484c-9c19-e7e324a39e21",
      "type": "demo.success",
      "payload": {
        "sleep_seconds": 1
      },
      "status": "queued",
      "priority": 50,
      "available_at": "2026-06-21T02:33:01.725170+03:30",
      "attempts": 0,
      "max_attempts": 3,
      "idempotency_key": null,
      "locked_by": null,
      "locked_until": null,
      "last_error": "",
      "result": null,
      "created_at": "2026-06-21T02:32:46.176967+03:30",
      "updated_at": "2026-06-21T02:32:46.176967+03:30",
      "finished_at": null
    }
  ]
}
```

### Get Job Detail

```http
GET /api/jobs/{id}/
```

Returns the same job shape as the list result item.

### Get Events For One Job

```http
GET /api/jobs/{id}/events/
```

Returns events ordered by ascending id:

```json
[
  {
    "id": 1,
    "job": "8421aca2-7d0c-484c-9c19-e7e324a39e21",
    "type": "created",
    "attempt": 0,
    "worker_id": null,
    "message": "Job created",
    "data": {
      "type": "demo.success"
    },
    "created_at": "2026-06-21T02:32:46.176967+03:30"
  }
]
```

### Global Event Feed

```http
GET /api/job-events/?limit=50&type=failed&job_id={uuid}
```

Pagination:

- cursor pagination
- ordering: newest first by `id`
- default page size: `50`
- max page size: `200`

Filters:

| Query param | Behavior |
| --- | --- |
| `type` | Exact event type match. |
| `job_id` | Events for one job. |

Response:

```json
{
  "next": "http://localhost:8000/api/job-events/?cursor=cD05NzM%3D&limit=50",
  "previous": null,
  "results": [
    {
      "id": 973,
      "job": "8421aca2-7d0c-484c-9c19-e7e324a39e21",
      "type": "failed",
      "attempt": 3,
      "worker_id": "638b98e627f5:1:thread-1:4e3b5893",
      "message": "Intentional demo failure",
      "data": {
        "error": "Intentional demo failure"
      },
      "created_at": "2026-06-21T02:33:20.600665+03:30"
    }
  ]
}
```

Cursor pagination is used for the event feed because events are append-only and live. Offset pagination can drift when new rows are inserted while the user is paging.

### Job Stats

```http
GET /api/jobs/stats/
```

Response:

```json
{
  "total": 53,
  "by_status": {
    "queued": 0,
    "running": 0,
    "succeeded": 41,
    "failed": 12
  },
  "overdue_running": 0,
  "retries_pending": 0,
  "oldest_queued_at": null
}
```

### Retry Failed Job

```http
POST /api/jobs/{id}/retry/
```

Only failed jobs can be manually retried.

Behavior:

```text
failed -> queued
attempts = 0
available_at = now
lock fields cleared
last_error cleared
result cleared
finished_at cleared
event = manual_retry
```

---

## Built-In Demo Job Types

Handlers are registered in `jobs/handlers.py`.

| Type | Behavior |
| --- | --- |
| `demo.success` | Sleeps briefly, renews lease, emits progress, succeeds. |
| `demo.fail` | Emits progress and raises an intentional error. |
| `demo.slow` | Sleeps longer, renews lease, emits progress, succeeds. |
| `demo.timeout` | Sleeps without lease renewal to demonstrate expired lease cleanup. |
| `demo.flaky` | Fails until a configured attempt, then succeeds. |

Example payloads:

```json
{
  "sleep_seconds": 3
}
```

```json
{
  "error": "Intentional demo failure"
}
```

```json
{
  "fail_until_attempt": 2
}
```

---

## Django Admin

The project uses `django-unfold` for the Django admin UI.

URL:

```text
http://localhost:8000/admin/
```

Create an admin user:

```powershell
docker compose exec backend python manage.py createsuperuser
```

The admin is useful for inspecting raw job state, event rows, locks, retry timestamps, and failed jobs.

---

## Local Development

Install dependencies in the backend environment:

```powershell
cd backend
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
```

Run migrations:

```powershell
.\.venv\Scripts\python.exe manage.py migrate
```

Run API:

```powershell
.\.venv\Scripts\python.exe manage.py runserver 0.0.0.0:8000
```

Run workers:

```powershell
.\.venv\Scripts\python.exe manage.py run_job_workers
```

Run the full Docker stack from the repository root:

```powershell
docker compose up --build
```

Rebuild only backend and worker:

```powershell
docker compose up -d --build backend worker
```

Stop all Docker services:

```powershell
docker compose down
```

---

## Testing

Run backend checks:

```powershell
cd backend
.\.venv\Scripts\python.exe -m ruff check .
$env:TEST_DATABASE_ENGINE='sqlite'; .\.venv\Scripts\python.exe -m pytest
```

Run backend tests with coverage:

```powershell
cd backend
$env:TEST_DATABASE_ENGINE='sqlite'; .\.venv\Scripts\python.exe -m pytest --cov --cov-report=term-missing
```

The test suite covers:

- successful claim
- concurrent claim prevention on PostgreSQL
- deterministic priority and `available_at` ordering
- idempotent create
- retry scheduling with exponential backoff
- exhausted attempts becoming failed
- expired lease cleanup
- stale worker cannot complete old attempt
- lease renewal ownership checks
- manual retry
- worker shutdown flag
- jobs limit-offset pagination
- global event cursor pagination

The PostgreSQL-specific concurrent claim test is skipped under SQLite because `SKIP LOCKED` is PostgreSQL behavior.

---

## Assumptions And Simplifications

This backend intentionally chooses a small, defensible design for the interview task. The following are explicit assumptions and simplifications:

- **Single internal queue**: there is no `Queue` table because queue routing is not part of the assignment. Job selection is controlled by `priority`, `available_at`, and deterministic ordering.
- **Ephemeral workers**: there is no `Worker` table because workers self-identify through generated `locked_by` values and `JobEvent.worker_id`. Persisting worker rows would require registration, heartbeat cleanup, and stale-row management.
- **No execution table**: `JobEvent` is enough for demo visibility and debugging. A larger production system may keep immutable per-attempt execution rows for duration metrics, artifacts, and richer attempt history.
- **No cancellation state**: cancellation is omitted to keep the state machine small and focused on `queued`, `running`, `succeeded`, and `failed`.
- **No batch claiming**: workers claim one job at a time. This keeps correctness and tests easier to reason about, although batch claiming can improve throughput.
- **No WebSockets or PostgreSQL `LISTEN/NOTIFY`**: the UI uses polling and cursor pagination for deterministic demo behavior.
- **No authentication**: the API is local/demo focused. Production would require authentication, authorization, and role-based controls for write/operator actions.
- **No retention policy**: `JobEvent` grows forever in this version. Production should archive, partition, or delete old events.
- **Clock assumption**: lease expiry depends on database and application clocks being reasonably synchronized.
- **Handler idempotency assumption**: external side effects must tolerate retries because delivery is at least once.

---

## Engineering Concepts Demonstrated

The implementation is small, but it deliberately demonstrates production-relevant backend concepts:

- transactional state machine
- row-level locking
- `SELECT ... FOR UPDATE SKIP LOCKED`
- partial indexes for hot queue queries
- database check constraints for valid row shape
- idempotency key for duplicate create protection
- ownership guard using `locked_by + attempts`
- lease-based failure recovery
- automatic exponential backoff
- at-least-once delivery semantics
- cursor pagination for append-only event feeds
- limit-offset pagination for bounded job lists
- append-only audit log
- graceful worker shutdown
- environment-based runtime configuration

---

## What Would Change For A Larger Production System

The current design is intentionally minimal. A larger production queue would likely add:

- Redis or a dedicated broker for very high throughput and broader distributed workloads.
- Authentication and authorization for all write endpoints and operator actions.
- Cancellation or cooperative stop states for jobs that should no longer run.
- Dead-letter metadata or a dead-letter inspection view for permanently failed jobs.
- Metrics, alerting, and dashboards for queue depth, job age, throughput, failures, and retries.
- Event retention, archival, or PostgreSQL partitioning for `JobEvent`.
- A dedicated execution table if detailed per-attempt history is required.
- Queue pause, drain, and resume controls for operational maintenance.
- Batch claiming if single-job claims become a throughput bottleneck.

---

## Operational Notes

### At-Least-Once Delivery

This queue provides at-least-once execution.

It prevents two active workers from owning the same job attempt. It does not guarantee that a handler's external side effects happen exactly once.

Example:

```text
1. Worker performs an external API call.
2. Worker process crashes before marking the job succeeded.
3. Lease expires.
4. Another worker retries the job.
```

The external API call may happen twice unless the handler is idempotent.

### Why PostgreSQL

PostgreSQL is used because the assignment focuses on deterministic transactional state and safe concurrent claiming.

`SELECT ... FOR UPDATE SKIP LOCKED` gives a simple and reliable concurrency primitive for a small internal queue:

- transactional claim
- row-level ownership
- no separate broker
- easy inspection through SQL/admin/UI

For a high-throughput distributed production queue, Redis-backed tools such as BullMQ or Sidekiq are often better. This project intentionally stays PostgreSQL-first for interview clarity and fewer moving parts.

### Why No Queue Model

There is one internal logical queue. `type`, `priority`, and `available_at` are enough for this demo.

Adding queue CRUD would add API surface and database complexity without improving the core correctness demonstration.

### Why No Worker Model

Workers are ephemeral. A worker id is generated at process/thread startup and stored only in `locked_by` and events.

A worker table would require registration, heartbeat cleanup, stale row management, and admin decisions that are not needed for this assignment.

### Why No Execution Model

`Job.attempts` stores the current attempt count, and `JobEvent` stores the audit timeline.

That is enough for this interview implementation. A separate immutable `JobExecution` table would be useful if the system needed detailed duration metrics, per-attempt artifacts, or long-term attempt history beyond events.

---

## Important Files

| File | Purpose |
| --- | --- |
| `config/settings.py` | Django, database, DRF, Unfold, and worker settings. |
| `jobs/models.py` | `Job` and `JobEvent` models, constraints, indexes. |
| `jobs/services.py` | Queue state machine and transactional service functions. |
| `jobs/handlers.py` | Demo job handlers. |
| `jobs/worker.py` | Threaded worker runner. |
| `jobs/management/commands/run_job_workers.py` | Worker management command and logging setup. |
| `jobs/views.py` | DRF API views. |
| `jobs/serializers.py` | API serializers. |
| `jobs/admin.py` | Django Unfold admin registration. |
| `jobs/tests/` | Backend tests. |

---

## Current API Summary

```http
GET  /api/health/
GET  /api/config/
POST /api/jobs/
GET  /api/jobs/?limit=25&offset=0&status=queued&type=demo
GET  /api/jobs/{id}/
GET  /api/jobs/{id}/events/
GET  /api/job-events/?limit=50&type=failed&job_id={uuid}
GET  /api/jobs/stats/
POST /api/jobs/{id}/retry/
GET  /api/schema/
GET  /api/docs/
```