Task Queues and Background Jobs

What belongs in a background job

• Sending emails, SMS and push notifications.
• Resizing images, transcoding video, generating PDFs.
• Calling slow or flaky third-party APIs (shipping rates, CRM sync).
• Imports and exports of large files, data clean-ups, reports.
• Anything that may take more than a few hundred milliseconds and does not need to finish before the user sees a response.

A task queue is a message queue plus a convention: each message names a task and its arguments, and a pool of workers runs them. Common tools: Celery (Python), Sidekiq (Ruby), BullMQ (Node.js), Laravel queues, Spring with RabbitMQ or SQS, and cloud services such as Cloud Tasks.

Long jobs: return a job ID

Retries and dead letters

Workers crash, APIs time out, databases restart. A job runner should retry failed jobs with growing delays (backoff), give up after a limit, and keep the failures somewhere visible.

import java.util.*;
import java.util.function.Predicate;

class JobRunner {
    // handler returns true on success. Returns [done, deadLetters].
    static List<List<String>> run(List<String> jobs, Predicate<String> handler, int maxAttempts) {
        Deque<String> queue = new ArrayDeque<>(jobs);
        Map<String, Integer> attempts = new HashMap<>();
        List<String> done = new ArrayList<>();
        List<String> dead = new ArrayList<>();
        while (!queue.isEmpty()) {
            String job = queue.pollFirst();
            int n = attempts.merge(job, 1, Integer::sum);
            if (handler.test(job)) {
                done.add(job);
            } else if (n < maxAttempts) {
                queue.addLast(job); // retry later (a real runner waits with backoff)
            } else {
                dead.add(job);
            }
        }
        return List.of(done, dead);
    }
}

from collections import deque

def run_jobs(jobs, handler, max_attempts):
    """handler(job) returns True on success. Returns (done, dead_letters)."""
    queue = deque(jobs)
    attempts = {}
    done, dead = [], []
    while queue:
        job = queue.popleft()
        attempts[job] = attempts.get(job, 0) + 1
        if handler(job):
            done.append(job)
        elif attempts[job] < max_attempts:
            queue.append(job)  # retry later (a real runner waits with backoff)
        else:
            dead.append(job)
    return done, dead

Scheduled jobs

Nightly reports, hourly clean-ups and subscription renewals run on a schedule. With several servers, a plain crontab on each one runs the job several times. Use one scheduler (a cron service, the task queue's scheduler, or a distributed lock so only one instance fires), and make the job safe to run twice anyway.

30 2 * * *   generate-daily-sales-report

Common mistakes

• Doing slow work inside the web request, so requests time out under load.
• Non-idempotent jobs that double-send or double-charge when retried.
• Retrying forever with no backoff, hammering a service that is already down.
• Running the same cron job on every server.

Interview questions

Try answering out loud before you open each one.

Check yourself

Related topics

• Message Queues and Delivery GuaranteesAsynchronism
• Back PressureAsynchronism
• Blob and Object StorageDatabases

Further reading

• Celery: Introduction Celery documentation
• Amazon SQS dead-letter queues AWS documentation
• Asynchronous Request-Reply pattern Microsoft Azure Architecture Center