// Foundations of Scale
Caching and the CDN
System Design track · 20 min
- Walk through the cache-aside pattern for a read, a miss and a write.
- Choose a TTL on purpose and protect a hot key from a stampede.
- Decide when a CDN should pull files from your server and when you should push them.
Your database answers a heavy query in 50 milliseconds. A cache in memory hands back the same answer in well under one millisecond, fifty to a hundred times faster.
Most slow systems you have ever used were slow for one simple reason: they kept asking the database questions it had already answered a minute ago.
Almost every system reads far more than it writes, and most of those reads are repeats: the same user profile, the same product page, the same top-ten list.
Sending every repeat to the database wastes its most precious resource. At small scale that is just slow. At large scale the database chokes, and the whole product goes down with it.
Think of a pharmacist. The medicines people ask for all day sit on the shelf right behind the counter. The rare ones stay in the storeroom at the back. Most customers are served in seconds, and the trip to the back is only for the unusual request.
A cache is that shelf. It is a bet that some data is read far more often than it changes. When the answer is on the shelf, that is a cache hit; when it is not, a miss, and you walk to the storeroom: the database. Make the bet only where the odds are good, and always know what a stale answer would cost you.
- On a read: look in the cache first. If the data is there (a hit), return it. The database is never touched.
- On a miss: read from the database, return the result, and put a copy in the cache. The next reader gets the fast path.
- On a write: update the database first, then delete the cached copy. The next read misses once, loads the fresh value and refills the cache.
In cache-aside your app is in charge: the cache is a simple store, and the app decides what goes in and when it leaves. Three other patterns move that job around.
Read-through: the cache loads from the database by itself on a miss, so your code only talks to the cache. Simpler code, but the cache now has to know your data source.
Write-through: every write passes through the cache to the database before it is confirmed. The cache is always fresh, but every write waits for the database.
Write-behind: the cache confirms the write at once and saves to the database later, in batches. Writes become very fast, and if the cache machine dies before saving, that data is lost. Speed, freshness and safety: you never get all three for free.
Every cached value has a TTL (time to live): how long it stays before it expires. A short TTL means fresher data and more load on the database. A long TTL means the opposite. You choose where to sit.
When the cache is full, an eviction policy decides what to throw out. The common one, LRU, removes whatever was used least recently.
Now the danger. A famous account's profile expires right at peak time. Thousands of requests miss at the same moment and all rush to the database together: a stampede (often called a thundering herd). Two fixes: spread the expiry times a little so keys don't expire together, or use a lock so only one request reloads the value while the others wait a moment.
Spreading the cache over many machines shares the load, until one key becomes enormous. A celebrity's post or a flash-sale product can draw millions of requests per second, and all of them land on the same cache machine. Adding machines does not help a single key.
The fix is a tiny local cache inside each app server, just for that handful of very hot keys, so the reads never leave the server. The price: hundreds of servers each hold their own copy, and no update reaches them, so each copy can be a little out of date until its very short TTL runs out.
Use it only where a few seconds of disagreement is harmless, like a view counter. Never for an account balance.
Static files, such as images, videos and JavaScript bundles, should not travel from your one data centre to every user on earth. A CDN (content delivery network) keeps copies of them on servers in many cities, called edge servers, and serves each user from the closest one.
The CDN respects the cache headers you set, including how long each file may be kept. If you must replace a file before its time is up, you tell the CDN to drop it (purge it). The idea is the same as a cache; what is new is the geography.
A push setup is like a warehouse you stock yourself. You upload files to the CDN, and every edge serves your copy until you replace it. You control exactly what users see, and your own server carries no traffic. It suits large files that rarely change, like videos and installers. The cost: publishing is now your job.
A pull setup is like a shop that orders stock when a customer asks. The first time a city's edge server is asked for a file, it fetches it from your server, keeps it and serves the repeats. Setup is almost nothing: point the CDN at your server and you are done. But the first user in each region waits for the slow trip, and your server must stay up to refill the caches.
Push for control, pull for convenience.
Never treat the cache as the source of truth. Caches lose data, expire entries and get wiped by a restart. The database holds the truth; the cache is a shortcut.
Any write path that updates the cache but not the database is a data-loss bug waiting for its moment.
In cache-aside, why does a write delete the cached copy instead of updating it? And what goes wrong if you update the cache but forget the database?
Answer in your head first, then check.
You host a 2 GB app installer that changes once a month, downloaded worldwide. Push zone or pull zone?
Answer in your head first, then check.
Memory beats disk by a wide margin, so answer repeats from memory: check the cache, on a miss read the database and fill the cache, and on a write update the database and delete the copy. Pick your TTLs on purpose, protect hot keys from stampedes, and let a CDN play the same trick for static files close to your users.
Find the three most repeated reads in a system you own. For each one, write down how often it changes, what a stale answer would cost, and what TTL you could live with. That is your caching shortlist.
- Walk through the cache-aside pattern for a read, a miss and a write.
- Choose a TTL on purpose and protect a hot key from a stampede.
- Decide when a CDN should pull files from your server and when you should push them.
Foundations of Scale