Distributed Data Primitives
Design a rate limiter
One misbehaving client — a broken retry loop, a scraper, an attacker — can spend your entire capacity on itself. The rate limiter is the bouncer at the door: everyone gets served, no one gets the whole bar.
You rate-limit for three reasons: protect the service from overload, keep usage fair between clients, and control cost when each request burns money. Enforce it at the API gateway or middleware — never in the client. Client-side limits are a suggestion; anything a user controls, a user can bypass.
- Fixed-window counter: count requests per user per minute window. Dead simple — but bursty at the edges: a client can fire 2× the limit by hitting the end of one window and the start of the next.
- Sliding-window log or counter: track timestamps (or blend two adjacent windows) so the window moves with time. Accurate at the edges — but heavier: more memory, more math per request.
- Token bucket: a bucket fills with tokens at a fixed rate; each request spends one. A full bucket allows a smooth burst, then the refill rate takes over. This is what most real APIs use.
- Leaky bucket: requests enter a queue and leave at a fixed rate, like water through a hole. Bursts get smoothed into a perfectly steady outflow — great when the thing behind you demands a constant pace.
Both buckets throttle, but the output shape is different. A token bucket lets a burst through — up to the bucket size — then throttles to the refill rate: the outflow is spiky, with a bounded average. A leaky bucket emits a perfectly constant stream no matter how ragged the input; excess requests wait in the queue or get dropped. Choose by what downstream tolerates: a payment gateway that chokes above 50 transactions per second wants the leaky bucket; a search API that absorbs bursts wants the token bucket. In interviews, name the shape you need, not just the algorithm.
The sliding-window log is exact but stores a timestamp per request — expensive at millions of requests per second. The sliding-window counter approximates it: keep counts for the current and previous fixed windows, then weight the previous window by its overlap with the rolling span. If the window is 60 seconds and you are 15 seconds in, estimate = current + previous × (45/60). You get the memory footprint of a fixed window with accuracy near the log — Cloudflare reported this cut the fixed window's edge error to a few percent. The trade: it assumes requests spread evenly across the previous window, which a concentrated burst violates.
Rules look like “100 requests per second per API key” or “5 login attempts per minute per IP” — pick the identity (user, IP, key) that matches the abuse you fear. When a client crosses the line, answer 429 Too Many Requests with a Retry-After header so well-behaved clients back off gracefully. Now the hard part: with ten limiter servers, each sees only a tenth of the traffic — counting locally undercounts by design. The standard fix is shared state in a fast central store (like Redis), with atomic increments to survive the race conditions when two servers judge the same request at the same instant.
The rules themselves should not be hardcoded. A config service holds them — per-endpoint quotas, per-plan tiers — and the limiter fleet pulls and caches them, so raising a limit rolls out in seconds without a deploy. Now the failure question: the config store is down and a limiter cannot refresh. Fail closed — keep enforcing the last cached rules — and you may reject good traffic on a stale limit. Fail open — let everything through — and you lose protection exactly when your systems are stressed. A common compromise: cache the rules with a long TTL, fail closed while the cache holds, fail open only when nothing was ever cached. Decide it in writing, not during the outage.
A rate limiter is a valve: you are deciding whose request dies so the system lives. Every rule you write is a policy about fairness disguised as an algorithm.
Why does a fixed window let a client briefly get 2× its quota — and which two algorithms fix that edge, at what cost? (The window boundary resets the count, so end-of-window plus start-of-window bursts stack; sliding window fixes it with more memory per user, token bucket fixes it with refill-rate math and a burst allowance.)
Limit at the gateway, not the client. Choose the algorithm for the behavior you want — token bucket for smooth bursts, leaky bucket for a steady stream, fixed window when simple beats perfect — answer 429 with Retry-After, and solve the distributed counting problem with shared, atomic state.
Write the rate-limit policy for one API you own, in plain language: the identity, the quota, the window, the response, and what happens when the limiter itself goes down. Five sentences. If sentence five surprises you, it should — most teams never decide it.
Distributed Data Primitives