Throttling and Queue-Based Load Leveling
Traffic is spiky, but capacity is not. Throttling caps how much work each client or the whole system accepts; queue-based load leveling turns bursts into a steady stream. Learn when to use each and how they fit with autoscaling.
In a nutshell
- Throttling limits work accepted (per client, per feature or overall) so the system stays within capacity.
- Load leveling puts a queue in front of a fixed-capacity service so bursts wait instead of overwhelming it.
- Autoscaling adds capacity, but takes minutes; throttling and queues protect you meanwhile.
- Throttle lower-priority work first, and tell clients clearly (429 with
Retry-After).
Three ways to meet a spike
| Strategy | How | Trade-off |
|---|---|---|
| Autoscale | Add instances when load rises | Takes time; costs money; some parts (databases) cannot scale quickly |
| Throttle | Reject or degrade work above a limit | Some requests fail or get less |
| Level with a queue | Accept work into a queue, process at a steady rate | Results arrive later |
Real systems combine them: queue what can wait, throttle what cannot, and autoscale whatever can grow.
Throttling
Throttling uses the same algorithms as rate limiting (token buckets, sliding windows), but the goal is protecting capacity rather than enforcing fairness or quotas. Options when the limit is reached:
- Reject with 429 or 503 and
Retry-After. - Degrade: switch off optional features (recommendations, live counters) under load.
- Prioritise: paying customers and checkout before free-tier batch jobs and analytics.
Queue-based load leveling
A queue flattens the spike
Diagram as text
- Sale traffic → Queue: spike
- Queue → Workers: steady
- Workers → Database
This only works for work that can finish later (orders confirmed by email, video processing, notifications). Watch queue age: if the oldest message keeps getting older, average load is above capacity and the queue is just delaying failure.
Common mistakes
- Relying on autoscaling alone for sudden spikes.
- Queueing work whose user is waiting synchronously for the result.
- Throttling everything equally instead of shedding low-priority work first.
Interview questions
Try answering out loud before you open each one.
A flash sale brings 50× normal traffic for 10 minutes. How do you cope?
Pre-scale stateless tiers, serve product pages from CDN and cache, put a waiting room or queue in front of checkout, process orders through a queue at the rate inventory and payments can handle, reserve stock atomically, throttle non-essential features, and give clients clear 429 responses with Retry-After.
Check yourself
Related topics
Further reading
- Throttling pattern Microsoft Azure Architecture Center
- Queue-Based Load Leveling pattern Microsoft Azure Architecture Center
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.