Back Pressure
When producers send work faster than consumers can handle it, queues grow until something breaks. Back pressure pushes the slowdown back to the source. Learn bounded queues, load shedding and how to tell clients to slow down.
In a nutshell
- If work arrives faster than it is processed, an unbounded queue grows until memory runs out or latency becomes useless.
- Back pressure signals upstream to slow down: block the producer, reject with "busy", or reduce what you accept.
- Bound every queue and buffer, and decide what happens when it is full.
- Under overload, shed load early and cheaply (reject at the edge) instead of doing work that will time out anyway.
The problem with endless queues
A queue smooths short bursts. But if 1,200 jobs arrive per minute and workers finish 1,000, the backlog grows by 200 every minute, forever. Each new job waits longer; by the time it is processed the user has given up, and the work is wasted. Eventually the queue's memory or disk fills and everything fails at once.
Pushing back
A bounded queue pushes back
Diagram as text
- Producer → Queue: enqueue
- Queue → Workers
- Queue → 503 + Retry-After: full: reject
| When full… | Effect | Use when |
|---|---|---|
| Block the producer | The producer waits until there is space | Producer and consumer are in the same process or pipeline (streams, TCP flow control) |
| Reject (fail fast) | Return "busy" (HTTP 503 or 429) with Retry-After | Requests from clients who can retry later |
| Drop | Discard new (or oldest) items | Data where the latest value matters most: metrics, positions, presence |
| Degrade | Do less per request: skip recommendations, serve cached pages | Optional parts of a response |
TCP itself uses back pressure: the receiver advertises how much buffer it has left, and the sender slows down when it shrinks. Reactive stream libraries (Reactive Streams, Project Reactor, RxJava) let consumers request only as many items as they can handle.
Load shedding
An overloaded server that accepts every request gets slower for everyone, until every request times out and useful throughput drops to zero. Rejecting some requests quickly keeps the rest fast.
- Limit concurrency: cap in-flight requests per server; reject beyond that.
- Check deadlines: if a request has already waited longer than the client will wait, drop it instead of processing it.
- Shed by priority: keep checkout and login working, drop analytics and prefetches first.
- Shed at the edge: the load balancer or gateway can reject cheaply before work is done.
- Autoscale too: shedding protects you while new capacity starts; it does not replace adding capacity.
Common mistakes
- Unbounded in-memory queues and thread pools.
- Treating a growing queue as fine because nothing has crashed yet.
- Processing requests whose client has already given up.
- Rejecting requests without telling clients when to retry.
Interview questions
Try answering out loud before you open each one.
The queue backlog keeps growing. What do you do?
First confirm the cause: arrival rate above processing rate. Short term, add consumers (if the downstream can take it) and shed or defer low-priority work. Bound the queue and push back on producers with rejections and Retry-After. Long term, make processing faster, batch work, or partition the queue to scale out. Alert on queue age, not just length.
Check yourself
Related topics
Further reading
- Using load shedding to avoid overload Amazon Builders' Library
- Queue-Based Load Leveling pattern Microsoft Azure Architecture Center
- Reactive Streams reactive-streams.org
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.