diff --git a/backend/README.md b/backend/README.md index 6ca447f..d509c86 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,58 +1,1097 @@ -# Backend +# Backend README -Django REST Framework backend for the minimal queue. +This backend is a minimal production-aware PostgreSQL job queue implemented with Django and Django REST Framework. -The backend owns two models: +The project intentionally avoids building a general queue platform. There is no `Queue` model, no `Worker` model, no `Execution` model, no Redis abstraction, and no public worker claim endpoint. The important interview requirement is implemented directly: -- `Job` -- `JobEvent` +- deterministic job state transitions +- valid database state enforced by constraints +- safe concurrent claiming with PostgreSQL row locks +- retries with exponential backoff +- worker leases and expired-lease recovery +- append-only execution events for UI/demo visibility +- environment-configured multi-threaded workers -Workers do not call public claim endpoints. The worker command imports the same service functions that API views use for creation, retry, stats, and event reads. +The backend has only two queue-domain tables: -## API +- `jobs_job` +- `jobs_jobevent` -```http -GET /api/health/ -POST /api/jobs/ -GET /api/jobs/ -GET /api/jobs/{id}/ -GET /api/jobs/{id}/events/ -GET /api/job-events/?after_id=123&limit=100 -GET /api/jobs/stats/ -POST /api/jobs/{id}/retry/ -``` +All queue behavior is implemented in service functions under `jobs/services.py`. -## Admin +--- -The project uses `django-unfold` for the Django admin UI. +## High-Level Architecture ```text -http://localhost:8000/admin/ +PostgreSQL + | + | stores valid queue state and append-only events + | +Django REST API + | + | creates jobs, lists jobs, lists events, retries failed jobs, exposes stats + | +Worker command + | + | imports the same backend service code + | runs N worker threads configured by environment variables + | +Demo handlers ``` -Create a user: +The API process and worker process use the same codebase. In Docker Compose they are separate services using the same image: -```powershell -docker compose exec backend python manage.py createsuperuser +```text +backend -> python manage.py migrate && python manage.py runserver 0.0.0.0:8000 +worker -> python manage.py run_job_workers ``` -## Worker +The worker does not call HTTP endpoints to claim jobs. It imports `claim_next_job`, `complete_job`, `fail_job`, `renew_lease`, and `cleanup_expired_jobs` from `jobs/services.py`. + +--- + +## Design Scope + +Included: + +- PostgreSQL-backed internal queue +- one logical internal queue +- priority scheduling +- delayed eligibility through `available_at` +- idempotency keys for job creation +- deterministic retries with exponential backoff +- worker leases +- lease renewal for long-running handlers +- expired lease cleanup +- graceful worker shutdown +- DRF API for jobs/events/stats +- cursor pagination for global events +- limit-offset pagination for jobs +- Django Unfold admin + +Explicitly not included: + +- queue CRUD +- queue database table +- worker database table +- execution database table +- cancellation state +- batch claiming +- Redis implementation +- cron service +- WebSockets/PostgreSQL `LISTEN/NOTIFY` +- exactly-once execution guarantee + +This is an at-least-once queue. A worker can perform a side effect and then crash before marking the job as succeeded. For real external work, handlers must be idempotent. + +--- + +## Core Queue Semantics + +### Statuses + +`Job.status` is intentionally small: + +```text +queued +running +succeeded +failed +``` + +Supported transitions: + +```text +queued -> running +running -> succeeded +running -> queued retry after handler failure or lease expiry +running -> failed attempts exhausted +failed -> queued manual retry +``` + +Unsupported transitions are rejected by service-level ownership checks and by database shape constraints. + +### Ownership + +A worker owns a job only when all of these are true: + +```text +status = running +locked_by = worker_id +attempts = attempt_number_that_worker_claimed +locked_until is in the future +``` + +Completion, failure, progress, and lease renewal all verify the worker still owns the same attempt. This prevents a stale worker from overwriting the result of a later attempt. + +Example stale worker scenario: + +```text +1. Worker A claims job attempt 1. +2. Worker A hangs. +3. Lease expires. +4. Cleanup requeues the job. +5. Worker B claims attempt 2. +6. Worker A wakes up and tries to mark attempt 1 succeeded. +7. The update affects zero rows because attempts no longer matches. +``` + +This is one of the most important correctness properties in the project. + +--- + +## Database Model + +### `Job` + +`Job` is the source of truth for queue state. + +Important fields: + +| Field | Purpose | +| --- | --- | +| `id` | UUID primary key. | +| `type` | Handler routing key, for example `demo.success`. | +| `payload` | JSON input for the handler. | +| `status` | One of `queued`, `running`, `succeeded`, `failed`. | +| `priority` | Higher priority is claimed first. Default is `50`. | +| `available_at` | Earliest time the job can be claimed. Used for delayed jobs and retry backoff. | +| `attempts` | Number of claim attempts already made. | +| `max_attempts` | Maximum allowed attempts before final failure. | +| `idempotency_key` | Optional unique key to prevent duplicate job creation. | +| `locked_by` | Ephemeral worker identity for the current running attempt. | +| `locked_until` | Lease expiry time for the current running attempt. | +| `last_error` | Latest failure reason. | +| `result` | JSON result for succeeded jobs. | +| `finished_at` | Required for terminal statuses. | + +### `JobEvent` + +`JobEvent` is an append-only audit/event timeline for demo visibility and debugging. + +Events are not the source of truth for queue state. The `Job` row is the source of truth. Events explain how the job reached its current state. + +Supported event types: + +```text +created +claimed +progress +lease_renewed +succeeded +retry_scheduled +failed +timeout_requeued +manual_retry +``` + +--- + +## Database Constraints + +The model enforces valid job shape in the database. + +### Valid Status Shape + +`queued` jobs: + +```text +locked_by is null +locked_until is null +finished_at is null +``` + +`running` jobs: + +```text +locked_by is not null +locked_until is not null +finished_at is null +``` + +`succeeded` and `failed` jobs: + +```text +locked_by is null +locked_until is null +finished_at is not null +``` + +### Attempts Constraint + +```text +attempts >= 0 +max_attempts >= 1 +attempts <= max_attempts +``` + +### Idempotency Constraint + +`idempotency_key` is unique only when present: + +```text +unique idempotency_key where idempotency_key is not null +``` + +This allows many jobs with no idempotency key while preventing duplicate client-created jobs when a key is supplied. + +--- + +## Indexes + +### Claim Index + +```text +job_claim_idx +fields: available_at, -priority, created_at, id +condition: status = queued +``` + +This supports the hot claim query: + +```text +WHERE status = queued + AND available_at <= now() + AND attempts < max_attempts +ORDER BY priority DESC, available_at ASC, created_at ASC, id ASC +``` + +### Timeout Index + +```text +job_timeout_idx +fields: locked_until +condition: status = running +``` + +This supports expired lease cleanup. + +### Event Indexes + +```text +job_event_job_id_idx: job, id +job_event_id_idx: id +``` + +These support job detail timelines and global cursor-paginated event feeds. + +--- + +## ERD DBML + +Copy this DBML into [dbdiagram.io](https://dbdiagram.io) to generate the ERD. + +```dbml +Project minimal_job_queue { + database_type: "PostgreSQL" + Note: ''' + Minimal PostgreSQL-backed job queue. + + The queue domain intentionally has two tables: + - jobs_job: current queue state and lock ownership + - jobs_jobevent: append-only audit/event timeline + + There is no queue table, worker table, execution table, or task table. + ''' +} + +Enum job_status { + queued + running + succeeded + failed +} + +Enum job_event_type { + created + claimed + progress + lease_renewed + succeeded + retry_scheduled + failed + timeout_requeued + manual_retry +} + +Table jobs_job { + id uuid [pk, not null, note: "UUID primary key generated by Django uuid.uuid4"] + + type varchar(120) [not null, note: "Handler routing key, for example demo.success"] + payload jsonb [not null, default: `'{}'::jsonb`, note: "Handler input payload"] + + status job_status [not null, default: "queued", note: "queued | running | succeeded | failed"] + priority smallint [not null, default: 50, note: "Higher priority is claimed first"] + available_at timestamptz [not null, note: "Earliest time the job can be claimed"] + + attempts integer [not null, default: 0, note: "Number of attempts already claimed"] + max_attempts integer [not null, default: 3, note: "Maximum attempts before final failure"] + + idempotency_key varchar(255) [unique, note: "Nullable unique key. Real DB uses a partial unique index where not null."] + + locked_by varchar(255) [note: "Ephemeral worker id for current running attempt"] + locked_until timestamptz [note: "Lease expiry for current running attempt"] + + last_error text [not null, default: "", note: "Latest handler or lease error"] + result jsonb [note: "JSON result after success"] + + created_at timestamptz [not null, note: "Created timestamp"] + updated_at timestamptz [not null, note: "Updated timestamp"] + finished_at timestamptz [note: "Set only for succeeded/failed jobs"] + + indexes { + status [name: "jobs_job_status_idx"] + available_at [name: "jobs_job_available_at_idx"] + created_at [name: "jobs_job_created_at_idx"] + (available_at, priority, created_at, id) [name: "job_claim_idx", note: "Partial index in PostgreSQL: WHERE status = 'queued'. priority is DESC in Django migration."] + locked_until [name: "job_timeout_idx", note: "Partial index in PostgreSQL: WHERE status = 'running'."] + idempotency_key [name: "job_idempotency_key_unique", unique, note: "Partial unique index in PostgreSQL: WHERE idempotency_key IS NOT NULL."] + } + + Note: ''' + Check constraints implemented in Django/PostgreSQL: + + job_status_valid_shape: + queued: + locked_by IS NULL + locked_until IS NULL + finished_at IS NULL + + running: + locked_by IS NOT NULL + locked_until IS NOT NULL + finished_at IS NULL + + succeeded/failed: + locked_by IS NULL + locked_until IS NULL + finished_at IS NOT NULL + + job_attempts_valid: + attempts >= 0 + max_attempts >= 1 + attempts <= max_attempts + ''' +} + +Table jobs_jobevent { + id bigint [pk, increment, not null] + job_id uuid [not null, ref: > jobs_job.id] + + type job_event_type [not null, note: "created | claimed | progress | lease_renewed | succeeded | retry_scheduled | failed | timeout_requeued | manual_retry"] + attempt integer [not null, default: 0] + worker_id varchar(255) [note: "Worker identity that emitted the event, if applicable"] + message text [not null, default: ""] + data jsonb [not null, default: `'{}'::jsonb`] + created_at timestamptz [not null] + + indexes { + type [name: "jobs_jobevent_type_idx"] + created_at [name: "jobs_jobevent_created_at_idx"] + (job_id, id) [name: "job_event_job_id_idx"] + id [name: "job_event_id_idx"] + } + + Note: ''' + Append-only audit timeline. + Job state is not derived from this table. + The jobs_job row remains the source of truth. + ''' +} +``` + +--- + +## Claiming Algorithm + +The worker claims a job in a short transaction: + +```python +queryset = Job.objects.filter( + status=Job.Status.QUEUED, + available_at__lte=now, + attempts__lt=F("max_attempts"), +).order_by("-priority", "available_at", "created_at", "id") + +queryset = queryset.select_for_update(skip_locked=True) +job = queryset.first() +``` + +Then the selected job is updated: + +```text +status = running +attempts = attempts + 1 +locked_by = worker_id +locked_until = now + lease_seconds +last_error = "" +result = null +``` + +And a `claimed` event is inserted. + +Why `SKIP LOCKED` matters: + +- multiple worker threads can query at the same time +- one worker locks a candidate row +- other workers skip locked rows instead of blocking +- each job attempt is claimed by at most one worker + +The handler runs outside the claim transaction. The database row is not held locked while work is executing. + +--- + +## Completion Algorithm + +Success is persisted only if the worker still owns the same attempt: + +```text +WHERE id = job_id + AND status = running + AND locked_by = worker_id + AND attempts = attempt +``` + +If the update affects zero rows, the worker lost ownership. The service returns `None` and does not force the job to succeeded. + +On success: + +```text +status = succeeded +result = handler result +locked_by = null +locked_until = null +finished_at = now +``` + +Then a `succeeded` event is inserted. + +--- + +## Failure And Retry Algorithm + +On handler failure, the worker calls `fail_job`. + +If attempts remain: + +```text +running -> queued +available_at = now + backoff_delay +locked_by = null +locked_until = null +last_error = error message +event = retry_scheduled +``` + +If attempts are exhausted: + +```text +running -> failed +locked_by = null +locked_until = null +finished_at = now +last_error = error message +event = failed +``` + +Backoff is deterministic: + +```text +delay_seconds = min(base_seconds * 2 ^ (attempt - 1), max_backoff_seconds) +``` + +With defaults: + +```text +attempt 1 -> 5 seconds +attempt 2 -> 10 seconds +attempt 3 -> 20 seconds +... +max delay -> 300 seconds +``` + +--- + +## Lease Renewal + +Long-running handlers can renew their lease: + +```python +context.renew() +``` + +Lease renewal succeeds only if: + +```text +id = job_id +status = running +locked_by = worker_id +attempts = attempt +``` + +This means a stale worker cannot renew an old attempt after the job has already been requeued and claimed by another worker. + +The demo `demo.slow` handler renews leases while it sleeps. + +The demo `demo.timeout` handler intentionally does not renew leases so the UI can demonstrate lease expiry and requeue/failure behavior. + +--- + +## Expired Lease Cleanup + +Every worker thread periodically runs cleanup: + +```text +status = running +locked_until < now +``` + +Expired jobs are locked with `select_for_update(skip_locked=True)`. + +If attempts remain: + +```text +running -> queued +available_at = now + backoff_delay +event = timeout_requeued +``` + +If attempts are exhausted: + +```text +running -> failed +event = failed +``` + +There is no separate cron process. This is intentional for the interview version: the worker process owns both execution and lease cleanup. + +--- + +## Worker Runtime + +Run workers with: ```powershell python manage.py run_job_workers ``` -Worker settings come from environment variables: +In Docker: -```env -JOB_WORKER_THREADS=4 -JOB_WORKER_POLL_INTERVAL_MS=500 -JOB_WORKER_LEASE_SECONDS=30 -JOB_WORKER_CLEANUP_INTERVAL_SECONDS=5 -JOB_WORKER_BACKOFF_BASE_SECONDS=5 -JOB_WORKER_BACKOFF_MAX_SECONDS=300 -JOB_WORKER_MAX_ATTEMPTS_DEFAULT=3 -JOB_WORKER_LOG_LEVEL=INFO +```powershell +docker compose up worker +``` + +The worker runner: + +1. starts `JOB_WORKER_THREADS` Python threads +2. gives each thread its own generated worker id +3. periodically cleans expired leases +4. claims one eligible job +5. executes the matching handler +6. marks the job succeeded, retry scheduled, or failed +7. sleeps when no job is available +8. stops claiming new jobs on `SIGINT`/`SIGTERM` + +Worker id format: + +```text +hostname:pid:thread-number:short-random-id +``` + +Example: + +```text +638b98e627f5:1:thread-1:4e3b5893 +``` + +--- + +## Environment Variables + +### Django/PostgreSQL + +| Variable | Default | Purpose | +| --- | --- | --- | +| `DEBUG` | `true` | Enables debug mode and permissive CORS in local development. | +| `DJANGO_SECRET_KEY` | local dev key | Django secret key. Override outside local development. | +| `DJANGO_ALLOWED_HOSTS` | `localhost,127.0.0.1,0.0.0.0` | Allowed hosts. | +| `DJANGO_CORS_ALLOWED_ORIGINS` | `http://localhost:5173,http://127.0.0.1:5173` | Frontend origins. | +| `POSTGRES_DB` | `job_queue` | Database name. | +| `POSTGRES_USER` | `postgres` | Database user. | +| `POSTGRES_PASSWORD` | `postgres` | Database password. | +| `POSTGRES_HOST` | `localhost` | Database host. In Docker this is `postgres`. | +| `POSTGRES_PORT` | `5432` | Database port. | +| `POSTGRES_CONN_MAX_AGE` | `0` | Django persistent DB connection age. | +| `TIME_ZONE` | `Asia/Tehran` | Django timezone. | + +### Worker + +| Variable | Default | Purpose | +| --- | --- | --- | +| `JOB_WORKER_THREADS` | `4` | Number of worker threads in the worker process. | +| `JOB_WORKER_POLL_INTERVAL_MS` | `500` | Sleep interval when no job is claimable. | +| `JOB_WORKER_LEASE_SECONDS` | `30` | Lock lease duration for a running attempt. | +| `JOB_WORKER_CLEANUP_INTERVAL_SECONDS` | `5` | How often each worker thread checks expired leases. | +| `JOB_WORKER_CLEANUP_BATCH_SIZE` | `100` | Max expired leases cleaned in one cleanup pass. | +| `JOB_WORKER_BACKOFF_BASE_SECONDS` | `5` | Base retry backoff. | +| `JOB_WORKER_BACKOFF_MAX_SECONDS` | `300` | Maximum retry delay. | +| `JOB_WORKER_MAX_ATTEMPTS_DEFAULT` | `3` | Default max attempts for jobs that do not specify one. | +| `JOB_WORKER_LOG_LEVEL` | `INFO` | Worker terminal log level. Use `DEBUG` for detailed polling/claim/execution logs. | + +--- + +## API + +Base URL in local development: + +```text +http://localhost:8000/api +``` + +### Health + +```http +GET /api/health/ +``` + +Response: + +```json +{ + "ok": true, + "database": "ok" +} +``` + +If the database cannot be reached: + +```json +{ + "ok": false, + "database": "error", + "detail": "..." +} +``` + +### Create Job + +```http +POST /api/jobs/ +``` + +Body: + +```json +{ + "type": "demo.success", + "payload": { + "sleep_seconds": 1 + }, + "priority": 50, + "available_at": null, + "max_attempts": 3, + "idempotency_key": "optional-client-key" +} +``` + +Behavior: + +- creates a `queued` job +- sets `available_at` to now when omitted/null +- emits a `created` event +- returns `201 Created` for a new job +- returns `200 OK` with the existing job when `idempotency_key` already exists + +### List Jobs + +```http +GET /api/jobs/?limit=25&offset=0&status=queued&type=demo +``` + +Pagination: + +- limit-offset pagination +- default limit: `25` +- max limit: `100` + +Filters: + +| Query param | Behavior | +| --- | --- | +| `status` | Exact status match. | +| `type` | Case-insensitive substring match against job type. | + +Response: + +```json +{ + "count": 53, + "next": "http://localhost:8000/api/jobs/?limit=25&offset=25", + "previous": null, + "results": [ + { + "id": "8421aca2-7d0c-484c-9c19-e7e324a39e21", + "type": "demo.success", + "payload": { + "sleep_seconds": 1 + }, + "status": "queued", + "priority": 50, + "available_at": "2026-06-21T02:33:01.725170+03:30", + "attempts": 0, + "max_attempts": 3, + "idempotency_key": null, + "locked_by": null, + "locked_until": null, + "last_error": "", + "result": null, + "created_at": "2026-06-21T02:32:46.176967+03:30", + "updated_at": "2026-06-21T02:32:46.176967+03:30", + "finished_at": null + } + ] +} +``` + +### Get Job Detail + +```http +GET /api/jobs/{id}/ +``` + +Returns the same job shape as the list result item. + +### Get Events For One Job + +```http +GET /api/jobs/{id}/events/ +``` + +Returns events ordered by ascending id: + +```json +[ + { + "id": 1, + "job": "8421aca2-7d0c-484c-9c19-e7e324a39e21", + "type": "created", + "attempt": 0, + "worker_id": null, + "message": "Job created", + "data": { + "type": "demo.success" + }, + "created_at": "2026-06-21T02:32:46.176967+03:30" + } +] +``` + +### Global Event Feed + +```http +GET /api/job-events/?limit=50&type=failed&job_id={uuid} +``` + +Pagination: + +- cursor pagination +- ordering: newest first by `id` +- default page size: `50` +- max page size: `200` + +Filters: + +| Query param | Behavior | +| --- | --- | +| `type` | Exact event type match. | +| `job_id` | Events for one job. | + +Response: + +```json +{ + "next": "http://localhost:8000/api/job-events/?cursor=cD05NzM%3D&limit=50", + "previous": null, + "results": [ + { + "id": 973, + "job": "8421aca2-7d0c-484c-9c19-e7e324a39e21", + "type": "failed", + "attempt": 3, + "worker_id": "638b98e627f5:1:thread-1:4e3b5893", + "message": "Intentional demo failure", + "data": { + "error": "Intentional demo failure" + }, + "created_at": "2026-06-21T02:33:20.600665+03:30" + } + ] +} +``` + +Cursor pagination is used for the event feed because events are append-only and live. Offset pagination can drift when new rows are inserted while the user is paging. + +### Job Stats + +```http +GET /api/jobs/stats/ +``` + +Response: + +```json +{ + "total": 53, + "by_status": { + "queued": 0, + "running": 0, + "succeeded": 41, + "failed": 12 + }, + "overdue_running": 0, + "retries_pending": 0, + "oldest_queued_at": null +} +``` + +### Retry Failed Job + +```http +POST /api/jobs/{id}/retry/ +``` + +Only failed jobs can be manually retried. + +Behavior: + +```text +failed -> queued +attempts = 0 +available_at = now +lock fields cleared +last_error cleared +result cleared +finished_at cleared +event = manual_retry +``` + +--- + +## Built-In Demo Job Types + +Handlers are registered in `jobs/handlers.py`. + +| Type | Behavior | +| --- | --- | +| `demo.success` | Sleeps briefly, renews lease, emits progress, succeeds. | +| `demo.fail` | Emits progress and raises an intentional error. | +| `demo.slow` | Sleeps longer, renews lease, emits progress, succeeds. | +| `demo.timeout` | Sleeps without lease renewal to demonstrate expired lease cleanup. | +| `demo.flaky` | Fails until a configured attempt, then succeeds. | + +Example payloads: + +```json +{ + "sleep_seconds": 3 +} +``` + +```json +{ + "error": "Intentional demo failure" +} +``` + +```json +{ + "fail_until_attempt": 2 +} +``` + +--- + +## Django Admin + +The project uses `django-unfold` for the Django admin UI. + +URL: + +```text +http://localhost:8000/admin/ +``` + +Create an admin user: + +```powershell +docker compose exec backend python manage.py createsuperuser +``` + +The admin is useful for inspecting raw job state, event rows, locks, retry timestamps, and failed jobs. + +--- + +## Local Development + +Install dependencies in the backend environment: + +```powershell +cd backend +python -m venv .venv +.\.venv\Scripts\python.exe -m pip install -r requirements.txt +``` + +Run migrations: + +```powershell +.\.venv\Scripts\python.exe manage.py migrate +``` + +Run API: + +```powershell +.\.venv\Scripts\python.exe manage.py runserver 0.0.0.0:8000 +``` + +Run workers: + +```powershell +.\.venv\Scripts\python.exe manage.py run_job_workers +``` + +Run the full Docker stack from the repository root: + +```powershell +docker compose up --build +``` + +Rebuild only backend and worker: + +```powershell +docker compose up -d --build backend worker +``` + +Stop all Docker services: + +```powershell +docker compose down +``` + +--- + +## Testing + +Run backend checks: + +```powershell +cd backend +.\.venv\Scripts\python.exe -m ruff check . +$env:TEST_DATABASE_ENGINE='sqlite'; .\.venv\Scripts\python.exe -m pytest +``` + +The test suite covers: + +- successful claim +- concurrent claim prevention on PostgreSQL +- deterministic priority and `available_at` ordering +- idempotent create +- retry scheduling with exponential backoff +- exhausted attempts becoming failed +- expired lease cleanup +- stale worker cannot complete old attempt +- lease renewal ownership checks +- manual retry +- worker shutdown flag +- jobs limit-offset pagination +- global event cursor pagination + +The PostgreSQL-specific concurrent claim test is skipped under SQLite because `SKIP LOCKED` is PostgreSQL behavior. + +--- + +## Operational Notes + +### At-Least-Once Delivery + +This queue provides at-least-once execution. + +It prevents two active workers from owning the same job attempt. It does not guarantee that a handler's external side effects happen exactly once. + +Example: + +```text +1. Worker performs an external API call. +2. Worker process crashes before marking the job succeeded. +3. Lease expires. +4. Another worker retries the job. +``` + +The external API call may happen twice unless the handler is idempotent. + +### Why PostgreSQL + +PostgreSQL is used because the assignment focuses on deterministic transactional state and safe concurrent claiming. + +`SELECT ... FOR UPDATE SKIP LOCKED` gives a simple and reliable concurrency primitive for a small internal queue: + +- transactional claim +- row-level ownership +- no separate broker +- easy inspection through SQL/admin/UI + +For a high-throughput distributed production queue, Redis-backed tools such as BullMQ or Sidekiq are often better. This project intentionally stays PostgreSQL-first for interview clarity and fewer moving parts. + +### Why No Queue Model + +There is one internal logical queue. `type`, `priority`, and `available_at` are enough for this demo. + +Adding queue CRUD would add API surface and database complexity without improving the core correctness demonstration. + +### Why No Worker Model + +Workers are ephemeral. A worker id is generated at process/thread startup and stored only in `locked_by` and events. + +A worker table would require registration, heartbeat cleanup, stale row management, and admin decisions that are not needed for this assignment. + +### Why No Execution Model + +`Job.attempts` stores the current attempt count, and `JobEvent` stores the audit timeline. + +That is enough for this interview implementation. A separate immutable `JobExecution` table would be useful if the system needed detailed duration metrics, per-attempt artifacts, or long-term attempt history beyond events. + +--- + +## Important Files + +| File | Purpose | +| --- | --- | +| `config/settings.py` | Django, database, DRF, Unfold, and worker settings. | +| `jobs/models.py` | `Job` and `JobEvent` models, constraints, indexes. | +| `jobs/services.py` | Queue state machine and transactional service functions. | +| `jobs/handlers.py` | Demo job handlers. | +| `jobs/worker.py` | Threaded worker runner. | +| `jobs/management/commands/run_job_workers.py` | Worker management command and logging setup. | +| `jobs/views.py` | DRF API views. | +| `jobs/serializers.py` | API serializers. | +| `jobs/admin.py` | Django Unfold admin registration. | +| `jobs/tests/` | Backend tests. | + +--- + +## Current API Summary + +```http +GET /api/health/ +GET /api/config/ +POST /api/jobs/ +GET /api/jobs/?limit=25&offset=0&status=queued&type=demo +GET /api/jobs/{id}/ +GET /api/jobs/{id}/events/ +GET /api/job-events/?limit=50&type=failed&job_id={uuid} +GET /api/jobs/stats/ +POST /api/jobs/{id}/retry/ +GET /api/schema/ +GET /api/docs/ ``` -Use `JOB_WORKER_LOG_LEVEL=DEBUG` to log every worker poll, claim, lease renewal, progress update, retry, failure, and completion in the worker terminal.