Background Jobs and Queues: The Backbone of Reliable Apps

Your API should never render a PDF, resize a video, or call a flaky third party while a user stares at a spinner. The moment work can happen later, it should happen later. Queues are how you make that promise without silently losing data.

Why every request handler eventually needs a queue

A synchronous request handler couples your user's experience to the slowest thing it touches. Payment provider having a bad day? Your checkout is having a bad day. Email API rate limiting you? Your signup flow is timing out. That coupling is the root cause behind a huge share of the outages we get called in to fix.

The fix is old and boring: accept the request, persist the intent, return fast, do the work in a background worker. The categories show up in almost every product we build:

• Transactional email and SMS
• Webhook delivery and third party syncs (CRMs, payment providers, ERPs)
• Image, video, and document processing
• Report generation and data exports
• Billing runs, invoice generation, dunning
• Search index updates and cache warming
• Scheduled work: nightly reconciliation, cleanup, digests

If your app does three or more of these inline, you do not have an architecture problem waiting to happen. You have one now.

What a queue actually buys you

A queue is a buffer with delivery semantics. That one sentence hides four distinct wins:

• Decoupling. Producers do not know or care which worker runs the job, or when.
• Backpressure absorption. Traffic spikes pile up in the queue instead of falling over your database. The backlog drains when the spike passes.
• Retry semantics. A failed job goes back on the queue with a policy. A failed HTTP call inside a request handler just fails.
• Independent scaling. Web tier scales on request volume. Workers scale on queue depth. Those curves rarely match, so separating them saves real money.

Delivery guarantees: the part everyone gets wrong

Every queue system gives you one of two honest guarantees:

At-most-once

The broker hands the message to a worker and forgets it. If the worker crashes mid job, the job is gone. Acceptable for cache warming. Unacceptable for billing.

At-least-once

The worker must acknowledge completion. If it crashes or the ack never arrives before a visibility timeout expires, the broker redelivers. This is what you want almost everywhere, and it has one non negotiable consequence: every job will eventually run twice. Not might. Will. A worker that finishes the work and dies before acking produces a duplicate delivery, and no broker can prevent it.

"Exactly-once delivery" as a transport guarantee is marketing. Exactly-once processing is real, but it lives in your code, not the broker. You build it with idempotency, covered below.

Picking a broker without overthinking it

The broker matters less than your job design, but here is the honest landscape.

Redis with BullMQ or Sidekiq

The default for most product teams. Low latency, delayed jobs, priorities, rate limiting, repeatable jobs, all built in. The tradeoff: Redis persistence is configurable, not absolute. Run it with AOF enabled and treat a Redis loss as a recoverable incident, not an impossibility. For Node stacks we reach for BullMQ first; for Rails, Sidekiq remains excellent.

RabbitMQ

A real AMQP broker: exchanges, routing keys, per queue TTLs, native dead letter exchanges, publisher confirms. Worth it when you need fanout routing or strict broker side delivery features. The cost is one more stateful system to operate and a steeper mental model.

SQS

If you are on AWS, SQS is the boring, correct answer for many workloads. Fully managed, effectively unlimited throughput, dead letter queues and visibility timeouts as first class primitives. Latency is higher than Redis (polling based), FIFO queues cap throughput per message group, and local development needs an emulator. Fine tradeoffs for most teams.

Kafka is (usually) not your job queue

Kafka is a replayable log for event streams. It shines for event driven integration between services, which is a different problem than "run this job once, retry on failure." Per message acknowledgment, delayed retries, and priorities are all awkward on Kafka. If you are choosing between the two, we broke down the streaming side in our guide to event driven architecture.

Postgres with SKIP LOCKED

The underrated option. If you already run Postgres and your volume is modest (in our experience, up to a few hundred jobs per second is comfortable on decent hardware), a jobs table gives you something no external broker can: transactional enqueue. The job commits or rolls back atomically with your business data.

UPDATE jobs SET status = 'running', locked_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued' AND run_at <= now()
  ORDER BY priority DESC, run_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

SKIP LOCKED lets many workers poll the same table without blocking each other. Tools like Graphile Worker, pg-boss, Oban (Elixir), and Solid Queue (Rails) package this pattern properly. Start here more often than you think.

Designing jobs that survive production

The broker is 20 percent of the work. Job design is the rest.

Make every job idempotent

Since at-least-once means duplicates, every handler must produce the same result whether it runs once or five times. Concrete techniques:

• Natural idempotency. "Set user status to active" is safe to repeat. Prefer absolute writes over relative ones ("set balance to X" is dangerous for other reasons; "apply ledger entry with unique ID" is the right shape).
• Idempotency keys. Record a unique key per logical operation and check it before doing side effects:

const done = await db.processedJobs.insert(
  { key: job.idempotencyKey },
  { onConflict: "ignore" }
);
if (!done.inserted) return; // duplicate delivery, exit clean
await chargeCustomer(job.data);

• Pass through keys to third parties. Stripe, and most serious payment APIs, accept an idempotency key per request. Use the job's key so even the external side effect deduplicates.

Retries with exponential backoff and jitter

Immediate retries hammer a struggling dependency at the worst possible moment. Back off exponentially and add jitter so a thousand failed jobs do not retry in the same second:

await queue.add("send-invoice", payload, {
  attempts: 6,
  backoff: { type: "exponential", delay: 3000 }, // 3s, 6s, 12s, 24s...
  removeOnComplete: 1000,
});

Distinguish error classes. A validation error will fail identically on attempt 50; fail it fast. A network timeout deserves the full retry schedule.

Timeouts, heartbeats, and stuck jobs

Every job needs a hard timeout. A worker that hangs on a dead TCP connection holds its lock forever and quietly strangles throughput. Set the broker's visibility timeout (or lock duration) above your worst legitimate runtime, add an in process timeout below it, and use lock renewal (heartbeats) for genuinely long jobs like video encoding.

Poison messages and dead letter queues

Some jobs will never succeed: malformed payload, deleted record, permanent upstream rejection. Without a cap they retry forever and clog the queue. The pattern:

• Cap attempts (5 to 10 is typical).
• Route exhausted jobs to a dead letter queue, never delete them.
• Alert on DLQ arrivals. A DLQ nobody watches is a data loss log.
• Build a redrive path so you can fix the bug and replay the DLQ.

Failure modes we see over and over

These are the incidents that actually page people.

• Backlog growth. Producers outpace consumers, oldest job age climbs, and "background" work becomes hours late. Autoscale workers on queue depth or oldest job age, not CPU. Workers waiting on I/O show low CPU while the backlog explodes.
• The non transactional enqueue. Code writes the order to the database, then enqueues the confirmation email, and crashes between the two. Or worse: enqueues inside a transaction that later rolls back, so the worker processes an order that does not exist. The fix is the transactional outbox: write the job intent to an outbox table in the same transaction as the business data, and let a relay move it to the broker. Postgres based queues get this for free.
• Fat payloads. Serializing a whole document into the message balloons broker memory and breaks when the schema changes mid deploy. Pass IDs, fetch fresh state in the worker (the claim check pattern).
• Hidden ordering assumptions. Parallel workers reorder everything. If job B must follow job A, encode that explicitly (chained enqueue, workflow state machine, or FIFO groups), or make handlers order independent.
• One queue for everything. A flood of cheap notification jobs starves a critical billing job. Split queues by priority and latency class, and give critical queues dedicated workers.
• Database contention. Fifty concurrent workers hammering the same tables can hurt the very API you were protecting. Cap concurrency per queue and watch lock waits; our database scaling patterns guide covers the downstream side of this.

Operating workers like a production system

Workers are a first class deployment target, not a sidecar.

• Graceful shutdown. On SIGTERM: stop claiming new jobs, finish in flight work within a deadline, then exit. Combined with at-least-once delivery, deploys become invisible. It is the same discipline behind any zero downtime deployment strategy.
• Concurrency tuning. I/O bound jobs (API calls, email) tolerate high concurrency per process. CPU bound jobs (image processing) want roughly one per core, in a separate worker pool.
• Version skew. During a deploy, old payloads meet new code. Version your payloads or keep handlers backward compatible for one release.

The four metrics that matter

1. Oldest job age per queue. The single best alerting signal. Depth alone lies: a deep queue draining fast is fine, a shallow queue stuck for an hour is an incident.
2. Failure and retry rate. A rising retry rate predicts an outage before users feel it.
3. Processing duration percentiles. p95 creeping up means a dependency is degrading.
4. DLQ arrivals. Every arrival is a bug or a data problem. Zero is the target.

Wire jobs into your tracing so a user request and the background work it spawned share a trace. We covered the plumbing in observability with logs, metrics, and traces.

A rollout checklist

Moving inline work to a queue, in order:

1. Pick the boring broker: Postgres based if volume allows and you want transactional enqueue, Redis with BullMQ or Sidekiq for general product work, SQS if you are all in on AWS.
2. Define the job contract: name, versioned payload of IDs (not blobs), idempotency key.
3. Make the handler idempotent and prove it by running it twice in a test.
4. Set attempts, exponential backoff with jitter, and a hard timeout per job type.
5. Configure a dead letter queue with an alert and a documented redrive procedure.
6. Use a transactional outbox (or a Postgres queue) wherever a job must match a database write.
7. Split queues by priority; give critical paths dedicated workers.
8. Implement graceful shutdown and verify it during a real deploy.
9. Dashboard oldest job age, failure rate, duration percentiles, DLQ count. Alert on age and DLQ.
10. Load test the failure path: kill workers mid job, force redeliveries, confirm no duplicates reach users and no jobs vanish.

Teams that skip steps 3, 5, and 10 always come back to them after an incident. Cheaper to do them first.

How Innovation T can help

Innovation T designs and builds this layer for a living: job architecture, broker selection, idempotent handlers, outbox patterns, worker autoscaling, and the observability to prove it all works. Whether you are on Node, Python, PHP, or a Postgres monolith that just needs SKIP LOCKED done right, our engineering team ships queues that survive Black Friday and bad deploys alike. See our software and cloud engineering services.

If your app is dropping webhooks, timing out on checkout, or running billing by hand, talk to us. We will audit your current pipeline and give you a concrete migration plan, usually within a week.