Background Jobs & Queues
The problem
Some work (sending an email, processing an uploaded video) takes too long to do while a user's HTTP request waits for a response — background jobs let a server respond immediately and complete the slow work separately, tracked through a queue instead of blocking the request.
Why now
Stacks and queues already established the FIFO structure this pattern depends on; interprocess communication already showed that separate processes can communicate through a shared channel. A job queue is exactly that combination — a queue-based channel between the fast, request-handling process and a separate, slower worker process.
Mental model
Instead of doing slow work inline, a request handler pushes a description of the work onto a queue and immediately responds 'accepted' — a separate worker process pulls jobs off that same queue whenever it's free and does the actual work, completely decoupled in time from the original request. This is the queue data structure doing exactly what it's for: preserving order while letting production and consumption happen at different, independent rates.
Requires
- Stacks & QueuesComputer Science Fundamentals
A job queue is literally the queue data structure, persisted and shared between processes instead of living in one function's local memory; the FIFO ordering guarantee is identical.
- Interprocess CommunicationOperating Systems
A background worker is typically a separate process from the web server, and the job queue is the IPC channel connecting them — the exact cross-process communication pattern already covered.
Unlocks
Used in
- Email sending, image/video processing, and report generation done asynchronously
- Job queue systems (BullMQ, Sidekiq, Celery) built specifically around this pattern
- Retry logic for jobs that fail, decoupled from the original request that triggered them
- Scheduled/recurring jobs (cron-like behavior) built on the same queue infrastructure
Projects
- Build a simple job queue (even an in-memory array-backed one) where an HTTP endpoint enqueues a job and a separate worker loop processes it asynchronously
- Add retry logic to a background job that fails intermittently, with a maximum retry count before it's marked permanently failed
Examples
- A signup endpoint responds immediately with 201 Created and enqueues a 'send welcome email' job rather than waiting on a slow email API mid-request
- A video upload endpoint returns instantly while a background worker transcodes the video separately, updating status the client can poll
Resources
Mastery checklist
- I can identify work that belongs in a background job rather than inline in a request handler
- I can build a basic producer/consumer job queue
- I can implement retry logic with a maximum attempt count for a failing job
- I can explain why decoupling slow work from the request/response cycle improves perceived performance