amiirkhl/tabdeal-job-queue
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add readme architecture diagrams

Browse Source
This commit is contained in:
Amirhossein Khalili
2026-06-21 11:10:18 +03:30
parent dd9567deb5
commit bd358f5e48
20 changed files with 65 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
README.md
View File

@@ -1,4 +1,4 @@
# Minimal Senior-Level Job Queue
# Job Queue
This is a deliberately small PostgreSQL-backed job queue for the interview assignment.
@@ -14,6 +14,41 @@ The important parts are:
- idempotent job creation
- demo UI with job state and event polling
## Architecture
![prompt for generating an svg image for a minimal PostgreSQL-backed job queue architecture showing React UI talking to Django API, Django API using PostgreSQL, and a separate Django worker process with N configured threads claiming jobs from PostgreSQL with SKIP LOCKED; use clean interview-project style, simple labeled boxes, directional arrows, and callouts for two tables jobs and job_events](assets/images/project-architecture.png)
There is no queue table and no worker table. Workers are ephemeral process threads with generated ids. The queue is internal and ordered by:
```text
priority DESC, available_at ASC, created_at ASC, id ASC
```
## Statuses
![prompt for generating an svg image for the job status state machine with four states queued, running, succeeded, failed; arrows queued to running, running to succeeded, running to queued for retry after failure or timeout, running to failed when attempts are exhausted, and failed to queued for manual retry; use clear color coding and small labels on each transition](assets/images/job-status-state-machine.png)
The database also validates row shape:
- queued jobs cannot have locks or finish timestamps
- running jobs must have a lock owner and lease deadline
- terminal jobs must have a finish timestamp and no lock
## At-Least-Once Execution
![prompt for generating an svg image for at-least-once job execution failure recovery showing worker A claims attempt 1, worker A crashes or lease expires, cleanup requeues the job, worker B claims attempt 2, and stale worker A cannot complete attempt 1 because ownership no longer matches; use a horizontal timeline with worker lanes and database state callouts](assets/images/at-least-once-lease-recovery.png)
This queue provides at-least-once execution, not exactly-once execution.
A worker can perform an external side effect and crash before marking a job succeeded. The lease will expire and the job can run again. Real handlers should therefore be idempotent.
## Why PostgreSQL
The assignment requires PostgreSQL, and PostgreSQL gives a compact solution for safe concurrent claiming through `SELECT ... FOR UPDATE SKIP LOCKED`. This keeps the implementation transactional, inspectable, and easy to demo.
For a high-throughput distributed production queue, Redis-backed systems such as BullMQ or Sidekiq-style designs are common. That is documented as the next architecture, not implemented here.
## Run
```powershell
@@ -60,52 +95,6 @@ docker compose up -d --build worker
docker compose logs -f worker
```
## Architecture
```text
React UI
|
Django API ---- PostgreSQL
|
Django worker process
|
N worker threads from env
```
There is no queue table and no worker table. Workers are ephemeral process threads with generated ids. The queue is internal and ordered by:
```text
priority DESC, available_at ASC, created_at ASC, id ASC
```
## Statuses
```text
queued -> running
running -> succeeded
running -> queued retry after failure or timeout
running -> failed attempts exhausted
failed -> queued manual retry
```
The database also validates row shape:
- queued jobs cannot have locks or finish timestamps
- running jobs must have a lock owner and lease deadline
- terminal jobs must have a finish timestamp and no lock
## At-Least-Once Execution
This queue provides at-least-once execution, not exactly-once execution.
A worker can perform an external side effect and crash before marking a job succeeded. The lease will expire and the job can run again. Real handlers should therefore be idempotent.
## Why PostgreSQL
The assignment requires PostgreSQL, and PostgreSQL gives a compact solution for safe concurrent claiming through `SELECT ... FOR UPDATE SKIP LOCKED`. This keeps the implementation transactional, inspectable, and easy to demo.
For a high-throughput distributed production queue, Redis-backed systems such as BullMQ or Sidekiq-style designs are common. That is documented as the next architecture, not implemented here.
## Useful Commands
Run backend tests locally with SQLite fallback: