System Design Interview: Design a Rate Limiter Service

In a nutshell

Enforce limits at the gateway, with rules loaded from configuration and counters in a shared, fast store.
Use a token bucket or sliding window counter, updated atomically (for example a Redis Lua script).
Keep added latency to about one round trip to the counter store; batch or approximate for very hot keys.
Decide fail-open vs fail-closed when the store is down, and always return 429 with Retry-After.

Requirements

Functional: limit requests per key (API key, user, IP) per rule (for example 100 per minute on /search, 5 per minute on /login); reject excess with 429; rules change without deploys.
Non-functional: adds at most a few milliseconds; accurate within a small margin across all gateway instances; must not take the API down if it fails.

Estimates

Assume 50,000 requests per second at peak across 20 gateway instances and 10 million active keys. Each check is one small read-modify-write in the counter store: 50,000 operations per second, well within a small Redis cluster. A token bucket per key needs about 16–32 bytes of data plus key overhead: on the order of a gigabyte for 10 million keys.

API

An internal check call and a rule definition

Example

allow(key="api_key:k_123", rule="search") -> { allowed: true, remaining: 57, reset_in_s: 41 }

rules:
  - name: search    match: "GET /search"   limit: 100 per 60s   key: api_key
  - name: login     match: "POST /login"   limit: 5 per 60s     key: ip

Rejected response: 429 Too Many Requests, Retry-After: 41, X-RateLimit-Remaining: 0

Data model

Per (rule, key): either two numbers for a token bucket (tokens, last_refill) or two counters for a sliding window counter (current and previous window). Stored in Redis with a TTL slightly longer than the window, so idle keys disappear on their own.

High-level design

Rate limiting at the gateway

All steps Every gateway instance checks the same counters. Rules come from a configuration service and are cached locally.

Diagram as text

A request reaches any gateway instance.
The limiter runs one atomic script in Redis for the request's key and rule.
If allowed, the request continues to the API; if not, the gateway returns 429.
Rules are pushed or polled from configuration and cached in each gateway.

Deep dives

Atomic updates

Reading the counter, deciding and writing back in separate calls races under concurrency. Run the whole token-bucket update as one Lua script in Redis (scripts run atomically), or use INCR with EXPIRE for fixed windows. The algorithms and tested code are in the Rate limiting topic.

Hot keys and latency

A single huge client can make one Redis key hot. Options: shard keys across the cluster (keys naturally spread by hash slot), keep a small local token allowance per gateway that is topped up from Redis in batches (slightly less exact, far fewer calls), or give the biggest clients dedicated limits.

When Redis is down

Fail open (allow) keeps the product working but removes protection; fail closed (reject) protects the backend but causes an outage. A practical middle: fail open with a conservative local in-memory limit per gateway as a backstop, and alert.

Bottlenecks and trade-offs

Choice	Gain	Cost
Central counters	Exact global limits	A network hop per request
Local counters only	Zero added latency	Limit multiplied by the number of instances
Local allowance + periodic sync	Few calls, nearly exact	Short bursts can exceed the limit slightly

The last five minutes

Common mistakes

Per-instance counters presented as a global limit.
Non-atomic read-then-write counter updates.
No plan for the counter store failing.
Limiting only by IP, punishing users behind shared networks.

Interview questions

Try answering out loud before you open each one.

How do you enforce limits across multiple regions?

Either keep a separate budget per region (simple, each region enforces a share of the limit), or sync counters between regions asynchronously and accept small overshoots. A single global counter store adds cross-region latency to every request, which is rarely worth it.

Your notes

Saved in this browser as you type.

Finished this topic?

Mark it read to track your progress, then carry on with the next one.

Next topic