Design a Rate Limiter — System Design Interview Walkthrough

A rate limiter interview hinges on four answers: which algorithm you pick, how you keep the count correct across many servers, how you scale past a single Redis, and how it behaves when things break. Get those right and you have shown the depth they are probing. This applies the standard system design framework to a deceptively small problem — with the diagrams I draw on the whiteboard and a live token bucket you can poke at.

A rate limiter caps how many requests a client can make in a window (e.g., 100 requests/minute per user). It protects services from abuse, accidental overload, and runaway costs — and it is a favorite because it is small enough to finish yet rich enough to expose whether you understand distributed state. The naive single-server version is trivial; the interview is entirely about what happens when there are fifty gateways and a million requests a second.

1. Requirements

Functional: allow N requests per client per time window; reject excess with HTTP 429 Too Many Requests (and a Retry-After header); support different limits per scope (user, IP, API key, endpoint).

Non-functional, quantified:

• Low latency — the limiter sits in front of every request, so its added cost must be small (target: a few milliseconds, ideally under ~10 ms at the 99th percentile).
• High availability — if the limiter is down, the API should not go down with it.
• Scale — assume a large API: on the order of 1 million requests/second at peak across all clients.
• Accuracy — "approximately correct" is acceptable. Briefly allowing a few extra requests during a failover is fine; blocking legitimate traffic or crashing is not. This is an AP choice: we favor availability over perfect global consistency.

2. Capacity estimation (it decides the architecture)

Do the math, because it forces the hard part into the open. At 1M requests/second, every request needs a limiter check. A single Redis instance handles roughly 100,000–200,000 simple operations per second, and a token-bucket check is a few operations, so call it ~50,000–100,000 checks/second per Redis node.

1,000,000 checks/s ÷ ~75,000 checks/s per node ≈ ~13 Redis shards (plus replicas).

That single number is the whole reason the design cannot be "one Redis." It tells you immediately that you need to shard the keyspace and, ideally, to avoid a network hop on every single request. We will come back to both.

3. The algorithm (the crux)

Lead with token bucket for most APIs: it is O(1) state per client (just tokens + lastRefillTime), naturally allows short bursts, and is the safest default in an interview. The token bucket is easiest to feel by using one — click fast to drain the bucket and watch the 429s; pause and watch it refill:

A small burst is absorbed (the bucket starts full), but the sustained rate can never exceed the refill rate. Reach for the sliding window counter when bursts must be forbidden. Its formula is worth knowing, because it is exact enough without storing every request:

estimated = current_window_count
          + previous_window_count × (overlap fraction of the previous window)

# Example: limit 7/min, 30s into the current minute, previous minute saw 5, current saw 3:
3 + 5 × 0.5 = 5.5  ≤ 7  → allow

Cloudflare measured this approximation at about 0.003% error over ~400 million requests — close enough that it runs their production edge limiter. Knowing why fixed-window is flawed (the boundary burst) and that the sliding-window counter fixes it cheaply is the detail that signals depth.

4. The API response

When a request is allowed, the limiter is invisible. When it is rejected, return a clear, machine-readable response so good clients back off instead of retrying blindly:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1719500000

Return the X-RateLimit-* headers on every response, not just 429s, so clients can self-throttle before they hit the wall — the convention GitHub and Stripe follow.

5. Where it lives

Run the limiter at the edge / API gateway or as middleware before your application logic — you want to reject excess traffic before it consumes real resources. Frame the placement as a progression:

• In-process per server (bad at scale): each server limits independently, so with N servers a client effectively gets N× the limit. Fine for one box; wrong for a fleet.
• Dedicated limiter service / shared gateway (good): all gateways consult one shared store, giving a single global limit.
• Gateway + local fast-path (great): the gateway enforces, but keeps a local allowance to avoid a round trip on most requests (Section 7).

6. Single-node correctness: the atomic Redis answer

A single server can rate-limit in memory trivially. The interview's first real question is: with 50 app servers, how do you enforce one global limit per user?

The answer is a centralized store (Redis) holding each client's token state, updated with one atomic operation per request. A naive read-modify-write (read tokens, decide, write back) races under concurrency — two servers both read "1 token left" and both allow. The fix is to refill and consume in a single atomic step, a small Lua script that Redis runs without interleaving:

-- KEYS[1] = bucket key; ARGV = now, refill_rate, capacity, cost
local state = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1]) or tonumber(ARGV[3])   -- start full
local ts     = tonumber(state[2]) or tonumber(ARGV[1])

local refill = (tonumber(ARGV[1]) - ts) * tonumber(ARGV[2])
tokens = math.min(tonumber(ARGV[3]), tokens + refill)     -- refill, capped

if tokens >= tonumber(ARGV[4]) then
  tokens = tokens - tonumber(ARGV[4])
  redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', ARGV[1])
  redis.call('PEXPIRE', KEYS[1], 3600000)  -- TTL so idle buckets don't leak
  return 1            -- allowed
end
return 0              -- denied

One round trip, no race. Here is the full request path the gateway runs on every call:

7. Scaling past one Redis (where the interview is won)

Section 2 told us one Redis tops out around ~75k checks/s, so at 1M req/s we need to scale the store and, ideally, stop hitting it on every request. Two moves:

Shard the keyspace. Partition clients across a Redis Cluster (16,384 hash slots) by a hash of the client key, so each client's bucket lives on exactly one shard and the load spreads evenly. Adding shards is consistent-hashing-style rebalancing; a client never needs two shards because its limit is independent of everyone else's. Give each shard a replica for failover.

Cut the per-request hop. Even sharded, a network round trip per request adds latency and load. The production answer is a local-first design: each gateway keeps a small local token allowance and reconciles with Redis periodically (e.g., lease 10 tokens at a time, sync every second). Most requests are decided in-memory with no Redis call; Redis becomes the coordinator, not the hot path. Pair it with sticky routing (hash the client to a gateway at the load balancer) so a given client mostly lands on one gateway, making its local count meaningful. The cost is a little accuracy — a client might briefly exceed the global limit during a sync window — which our requirements already said is acceptable. This is how edge limiters like Cloudflare's run: enforce locally, reconcile globally. (A simpler alternative many APIs ship first is a single sharded Redis with the atomic check above — Stripe's published limiter is centralized like that; reach for local allowances only when the per-request hop actually hurts.)

8. Multi-tier limits and what you're really protecting against

Real limiters enforce several limits at once and reject if any is exceeded — for example per user (fairness), per IP (abuse), per API key (billing tier), and a global ceiling (capacity protection). Run each check and apply the most restrictive result.

It helps to name the two different jobs, because they pull the design in different directions:

• Abuse / DDoS protection → key on IP (or coarser), lean toward fail-closed at the edge, and push enforcement as far out (L3/L4, CDN) as possible.
• Fairness / cost control → key on user or API key, lean toward fail-open (a paying customer should not be blocked by a cache blip), and enforce at the application gateway.

Saying which problem you are solving — and that a serious system solves both with layered limits — is a strong senior signal.

The keying pitfalls worth naming. Each key has a sharp edge, and calling them out is rare-enough to stand out:

• Per IP is blunt. A corporate NAT, a university, or a coffee shop hides thousands of users behind one address, so one bad actor throttles them all. And the reverse bites too: an attacker rotating through the vast IPv6 space balloons your key cardinality and can exhaust the store's memory — so cap tracked IPs or limit on a /64 prefix, not a full address.
• Per user is precise but only exists after authentication — and the login endpoint itself is the trap. Limit failed logins per account too aggressively and an attacker can lock a victim out on purpose by burning their quota. There, limit on IP + account together and prefer slowdowns/CAPTCHAs over a hard lockout.
• Hybrid (both at once) is what real systems run; the cost is more keys and more memory per request — which the multi-tier check above is already paying for.

9. Availability and durability

The limiter is on the critical path, so it must not be a single point of failure:

• Redis replication + automatic failover (Redis Sentinel or Cluster) so a node loss does not take limiting offline. Losing a few token counts on failover is acceptable — we only need approximate limits.
• Connection pooling from gateways to Redis, so you are not paying a TCP/TLS handshake (tens of milliseconds) per request.
• Multi-region: run an independent limiter stack per region. Global limits across regions are expensive and rarely worth it; most designs accept per-region limits and say so.
• Fail-open as the default when Redis is unreachable (with fail-closed reserved for sensitive endpoints), so a store outage degrades limiting rather than the whole API.

10. Failure modes interviewers probe

11. Follow-ups a senior interviewer asks next

• Dynamic rule configuration: limits change without a redeploy. Store rules in a config service; gateways either poll every ~30s (simple, slightly stale) or get pushed updates via something like ZooKeeper/etcd (fast, more moving parts). Name the propagation-delay trade-off.
• Concurrent-request limiting: rate is requests-per-window; some systems also cap in-flight requests (e.g., "at most 20 concurrent per user") — a different limiter (a semaphore) that Stripe runs alongside the rate limiter.
• Hard vs soft vs elastic limits: the default is a hard limit (reject at the cap). Some systems allow a soft overage — a small burst credit above the cap — or an elastic one: let a client exceed its limit while the system has spare capacity and claw it back under load, which is really load-shedding wearing the limiter's clothes. A useful lever to name; don't build it unless the problem asks.
• Load shedding & prioritization: under genuine overload, shed low-priority traffic first (background/test requests before user-critical writes). Rate limiting protects against one noisy client; load shedding protects the system as a whole.

12. Trade-offs and what to say at each level

• Boundary burst (fixed window) → sliding window counter or token bucket.
• Race conditions on shared state → one atomic Lua script, never read-then-write.
• One Redis ceiling → shard by client key; local allowance + sticky routing to cut the hop.
• Accuracy vs latency → approximate local limits buy speed; name it, don't pretend it's free.

Mid-level: token bucket + a centralized Redis with an atomic check + return 429 with Retry-After. Senior: the capacity math (throughput and the per-algorithm memory argument), sharding, multi-tier limits with the IP-vs-user keying pitfalls, fail-open vs fail-closed, and the hot-key and clock-skew failure modes. Staff+: the local-allowance/sticky-routing fast path, multi-region, dynamic rule propagation, load shedding (and elastic limits), and the explicit AP trade-off — driven without prompting.

Why it shows up so often

It is compact but it forces a real algorithm choice and a genuine distributed-systems problem: atomic shared state, then scaling that state past one node. Master the token-bucket-on-Redis answer and the sharded, local-first scaling story, and you have covered the version asked in the vast majority of interviews. For the broader method, see the framework; for a read-heavy counterpart, the URL shortener.

Built and battle-tested in production by engineers who have run limiters at scale — written by Amit Singh, Senior SDE at Amazon, Claude Certified Architect, and the instructor at AlgoEngineer. We drill distributed-systems questions like this with live mock interviews in our System Design course.