// Distributed Data Building Blocks
Design a rate limiter
System Design track · 20 min
- Compare fixed window, sliding window, token bucket and leaky bucket, and pick one for a real API.
- Design a limiter that counts correctly across many servers, with a shared store and atomic updates.
- Decide in advance what happens when the limiter itself fails.
One badly behaved client can eat your whole capacity: a broken retry loop, a scraper, an attacker. Everyone else waits while one caller burns through your servers.
A rate limiter caps how many requests each client may send in a given time. It is the doorman of your API: everyone gets in, nobody gets the whole building.
You limit for three reasons: protect the service from overload, keep usage fair between clients, and control cost when each request costs you money (an SMS, a paid AI call).
Put the limiter on the server side, at the API gateway or in middleware in front of your app. Never rely on the client. A limit inside the app or browser is only a suggestion, and anything the user controls, the user can bypass.
- Fixed window: count each user's requests per minute and reset at :00. Very simple. The weak spot is the edge: 100 requests at 12:00:59 and 100 more at 12:01:00 means 200 requests in two seconds, double the limit.
- Sliding window: the window moves with the clock, so “the last 60 seconds” always means the last 60 seconds. No edge burst. It costs more memory or more math per request.
- Token bucket: a bucket refills with tokens at a fixed rate, and each request spends one. A full bucket allows a short burst, then traffic passes at the refill rate. Most real APIs use this.
- Leaky bucket: requests join a queue and leave at a fixed pace, like water dripping from a hole. However ragged the input, the output is perfectly steady.
Both buckets limit traffic, but what comes out is different. A token bucket lets a burst through, up to the bucket size, then holds traffic to the refill rate. The output is spiky, with a capped average. A leaky bucket always sends a steady stream. Extra requests wait in the queue or get dropped.
So choose by what sits behind the limiter. A payment provider that fails above 50 transactions per second wants the leaky bucket. A search API that can absorb short bursts wants the token bucket. In an interview, say which output shape you need, not just the algorithm's name.
The exact sliding window keeps a timestamp for every request. At millions of requests per second, that is a lot of memory. The sliding-window counter is the cheap approximation.
Keep just two counts: the previous minute and the current minute. Say the limit is 100 per minute. The previous minute had 80 requests, the current one has 30 so far, and you are 15 seconds into the current minute. The rolling 60 seconds still covers 45 seconds of the previous minute, so:
estimate = 30 + 80 × (45/60) = 30 + 60 = 90. That is under 100, so the request passes.
You get the memory cost of a fixed window with accuracy close to the exact log. When Cloudflare measured this method on 400 million requests, only 0.003% were wrongly allowed or blocked. The catch: it assumes the previous minute's requests were spread evenly, which a sharp burst breaks.
Rules look like “100 requests per second per API key” or “5 login attempts per minute per IP”. Pick the identity (user, IP, API key) that matches the abuse you fear.
When a client crosses the line, answer 429 Too Many Requests with a Retry-After header, so polite clients know how long to wait before trying again.
Now the hard part. With ten limiter servers behind a load balancer, each one sees only a tenth of a client's traffic. Counting locally lets the client through at ten times the limit. The standard fix is a shared counter in a fast central store like Redis, updated with atomic increments. Atomic means two servers updating the same counter at the same instant cannot overwrite each other.
Do not hardcode the rules. Keep them in a config service (quotas per endpoint, tiers per plan), and let the limiters pull and cache them. Raising a limit then takes seconds, with no deploy.
Now the failure question: the config store is down. Fail closed means keep enforcing the last rules you cached, which may reject good traffic on an old limit. Fail open means let everything through, which removes protection exactly when your systems are under stress.
A common middle path: cache the rules with a long TTL (time to live), keep enforcing while the cache holds, and fail open only if nothing was ever cached. Decide this in writing before the outage, not during it.
- Set a small capacity, press “Burst ×10”, and watch the bucket empty and the 429 rejections appear.
- Turn on auto traffic and move the refill rate. Capacity sets how big a burst can be; the refill rate sets the steady pace.
Why can a fixed window let a client get double its quota for a moment, and which two algorithms close that gap, at what cost?
Answer in your head first, then check.
Limit on the server, never the client. Pick the algorithm for the behaviour you want: token bucket for bursts with a capped average, leaky bucket for a steady stream, fixed window when simple is enough. Answer 429 with Retry-After, count in a shared store with atomic updates, and decide your failure mode before you need it.
Write the rate-limit policy for one API you own, in plain language: the identity, the quota, the window, the response, and what happens when the limiter itself goes down. Five sentences. If the fifth one surprises you, that is normal. Most teams never decide it.
- Compare fixed window, sliding window, token bucket and leaky bucket, and pick one for a real API.
- Design a limiter that counts correctly across many servers, with a shared store and atomic updates.
- Decide in advance what happens when the limiter itself fails.
Distributed Data Building Blocks