Job queues in web applications: why you need them and how they work

A job queue lets everything slow or unreliable — sending emails, generating reports, processing files, calling external APIs — run in the background while the user gets a response immediately. The web request enqueues a job in milliseconds, and separate worker processes execute it, retry on failure and keep one slow operation from taking down the whole site.

Here is how queues work, what to put in them and how not to lose jobs.

Why does a web application need a job queue?

A web application needs a job queue to separate a fast response from slow or unreliable work. While everything runs inside the request, any delay in an external service becomes a delay for the user, and any failure becomes an error on screen.

What typically goes into a queue:

• sending emails, SMS and push notifications;
• processing uploads — images, video, documents;
• generating reports and exports;
• calling external APIs: payment providers, CRMs, delivery services;
• calls to AI models that take seconds to respond;
• recomputing aggregates and search indexes.

The rule is simple: if the operation is not needed to answer the user right now, it belongs in a queue.

How does a job queue work?

A job queue has three parts: the application enqueues a job, a broker stores it, and worker processes pick it up and run it. Between enqueueing and execution the job lives in storage and survives application restarts.

Part	What it does	Typical options
Producer	Puts a job with its data on the queue	Application code: controller, event, scheduler
Broker	Stores jobs until they run	Redis, the database, RabbitMQ, cloud queues
Worker	Takes a job and executes it	Separate processes, scaled with load
Failed job store	Keeps jobs that could not complete	A failed jobs table, a dead-letter queue
Monitoring	Shows queue lengths and errors	Framework dashboards, metrics and alerts

Most frameworks have queues built in: Laravel Queues, BullMQ for Node.js, Celery for Python. To start, Redis or even the database itself is usually enough as a broker.

Photo: Unsplash

How do you write a reliable background job?

A reliable background job is idempotent, time-limited and retryable: it can safely run twice, never hangs forever and retries with a delay after a temporary failure. Most queue problems come from breaking one of these rules.

A Laravel job that sends an order to a CRM:

final class SendOrderToCrm implements ShouldQueue
{
    use Queueable;

    public int $tries = 5;
    public int $timeout = 30;

    public function __construct(public int $orderId) {}

    /** Delay between attempts grows: 10 s, 1 min, 5 min… */
    public function backoff(): array
    {
        return [10, 60, 300, 900];
    }

    public function handle(CrmClient $crm): void
    {
        $order = Order::findOrFail($this->orderId);

        // Idempotency: a repeated run will not create a duplicate in the CRM
        if ($order->crm_id !== null) {
            return;
        }

        $order->update(['crm_id' => $crm->createDeal($order)]);
    }
}

// In the controller — instant; the user does not wait for the CRM
SendOrderToCrm::dispatch($order->id);

Rules that save headaches

1. Pass an ID into the job, not the whole object: data may change before the job runs.
2. Enqueue after the transaction commits, or the worker may not find a record that is not saved yet.
3. Split queues by priority: password reset emails must not wait behind a thousand reports.
4. Limit execution time and attempts — endless retries clog the queue.

How do you avoid losing jobs and notice failures?

Jobs are not lost when the broker stores them durably, failed attempts are kept separately, and queue length and error counts are monitored with alerts. The most dangerous situation is a silent one: workers crashed, jobs piled up, and nobody noticed until customers complained.

What to set up:

• automatic worker restarts through a process manager or orchestrator;
• restarting workers on every deploy so they pick up new code;
• a failed job store with regular review of causes;
• alerts on queue length and job wait time;
• alerts on rising errors for a specific job type.

When do you need a dedicated message broker?

A dedicated broker such as RabbitMQ or Kafka is needed when the queue becomes a channel between several services or event volume reaches thousands per second. For background jobs inside one application, Redis or the database is usually enough.

Signs it is time to look at a broker:

• one application enqueues jobs and another, on a different stack, runs them;
• several independent consumers must receive the same event;
• you need guaranteed delivery with acknowledgements and complex routing;
• you need to store an event stream and replay it.

This is the same kind of decision as monolith versus services — see monolith vs microservices. Queues are especially useful for AI model calls, as covered in how to integrate an LLM into your product.

In short

• Queue everything that is not needed to answer the user right now.
• A queue is a producer, a broker, workers, a failed job store and monitoring.
• Jobs must be idempotent, time-limited and retryable with backoff.
• Enqueue after commit, split queues by priority and watch their length.
• A dedicated broker is for communication between services and large event streams.