Background Jobs Explained: Queues, Workers and Retries

Background jobs are pieces of work your app does outside the web request: sending an email, resizing an image, generating a PDF report. The request saves a small note saying “do this”, answers the user at once, and a separate process does the work a moment later.

This guide explains how that works, when it is worth it, and the one rule most teams learn the hard way: a job can run more than once. We measured both the speed-up and the double run with a real queue.

On this page

• Why move work out of the request?
• The three parts: producer, queue, worker
• Triggered jobs and scheduled jobs
• A crashed worker means a second run
• Rules for jobs that survive production
• Choosing a job queue
• FAQ

Why move work out of the request?

Users wait for every millisecond of a web request. If signing up sends a welcome email through a provider that takes 2 seconds, the sign-up page takes 2 seconds too. If the provider is down, sign-up fails.

Put the email on a queue instead and the request only waits for the enqueue. With BullMQ 6.7 on a local Redis, adding a job took 0.65 ms at the median and 1.44 ms at the 99th percentile, over 200 jobs. Over a network, add one round trip.

Use a background job when the work is slow, can fail and be retried, or the user does not need the result in the same response.

The three parts: producer, queue, worker

• Producer: your web app. It adds a job with the data needed, such as a user ID, not the whole user.
• Queue: where jobs wait. Usually Redis or a Postgres table, so jobs survive a restart of your app.
• Worker: a separate process that takes jobs one by one, runs them, and marks each done or failed.

The worker is a long-running program with no web address. It is deployed separately from the web app, so a slow job never blocks a page, and you can run more workers when the queue grows.

Triggered jobs and scheduled jobs

Jobs start in two ways:

Triggered jobs need a queue and a worker. Scheduled jobs can be a plain cron job that starts, works and exits; our guide to what a cron job is covers that side. Many queues also accept delayed and repeating jobs, so one worker can handle both.

A crashed worker means a second run

We started a job that takes 10 seconds, then killed the worker process 3 seconds in, the way an out-of-memory kill or a deploy would, and started a second worker right after the kill.

BullMQ noticed the dead worker’s lock had expired and handed the job out again, about a minute later with default settings. That is the right behaviour, because the work was never finished. But the first 3 seconds of the job had already happened.

If those 3 seconds sent an email or charged a card, the customer now gets it twice. Every queue that retries works this way. It is called at-least-once delivery, and it is why the first rule below matters.

Rules for jobs that survive production

1. Make jobs idempotent. Running a job twice must do no extra harm. Record that the email was sent, or use a unique key with your payment provider, and check it before acting.

2. Pass IDs, not objects. Put userId: 42 in the job, and load fresh data when the job runs. The user may have changed their email since the job was queued.

3. Retry with backoff, then stop. Retry a failed job a few times with growing delays, then move it to a failed list a person can look at. Retrying forever hides the problem.

4. Keep jobs short. One long job holds a worker and is the most likely to be cut off by a deploy. Split a big import into many small jobs.

5. Watch the queue length. A queue that keeps growing means workers cannot keep up. That is your signal to add workers or speed the job up.

Choosing a job queue

Redis-backed queues are the most common and fast; Redis lists are the classic building block. A Postgres queue means one less service: workers take jobs with FOR UPDATE SKIP LOCKED, described in the PostgreSQL SELECT documentation, so two workers never take the same row.

We tested the two most common in depth: BullMQ with Redis and Celery with Redis, including how many Redis commands an idle worker sends.

Running background jobs on Voroa

On Voroa, a worker is a Background Worker: built from your GitHub repository, with no web address, restarted if it exits. Run your web app as a web service, the worker beside it, and a managed Redis or Postgres database as the queue, all in Mumbai.

A program that finishes and exits belongs in a Cron Job instead, or Voroa will keep restarting it. The guide worker or cron job helps you pick, and deploying a background worker has the steps. Plans and sizes are on the background workers page.

FAQ

What is a background job? Work your app does outside the web request, in a separate process, such as sending email or generating a report. The user gets a response without waiting for it.

What is the difference between a background job and a cron job? A background job is usually triggered by an event and picked up by a worker. A cron job is started by the clock. Many apps use both.

Can a background job run twice? Yes. If a worker crashes mid-job, the queue hands the job out again. In our test the second run started about a minute after the crash. Make jobs safe to repeat.

Do I need Redis for background jobs? No. Redis is the most common queue, but a Postgres table with SKIP LOCKED works well at small and medium scale.

How many workers do I need? Start with one. Add more when the queue length keeps growing or jobs wait too long to start.

Measured with BullMQ 6.7.1 on a local Redis on 7 November 2026. Found something out of date? Write to [email protected].