Product-Scale Systems
Design a news feed
A user posts a photo. A second later, half a billion people could see it on their home page — and every one of them expects it in under 200 milliseconds. The question that decides the whole architecture is deceptively small: when do you compute each person's feed?
- Publishing: a post is created and must reach the feeds of everyone who follows the author — this is the fanout problem.
- Reading: a user opens the home page and expects their feed assembled, ranked, and rendered fast — no matter how the publishing half works.
- Scale: ~500M posts per day (≈ 6,000/s), reads outnumber writes ~100:1, average user follows ~300 accounts, and a handful of celebrities carry tens of millions of followers each.
The moment a post is created, a fanout worker writes the post's ID into every follower's precomputed feed cache. Reading the home page then becomes one cheap lookup: fetch the cached list of post IDs, hydrate it, done. For an average author with 300 followers, a post costs 300 writes — trivial. But a celebrity with 50M followers turns one post into 50M cache writes. Even at 100,000 writes per second, that is eight minutes of write storm for a single selfie — and most of those followers will not open the app for days.
Flip the decision: write nothing on publish, compute everything at read time. A post costs exactly one write — the celebrity problem vanishes. But now every home-page load must fetch the recent posts of all ~300 accounts you follow, merge them, rank them, and return the top slice, all inside your latency budget. Multiply that heavy merge by hundreds of millions of daily page loads and you have traded a write storm for a permanent read-time inferno.
Push for normal users, pull for celebrities. The fanout worker checks the author's follower count: below a threshold (say 10K–1M followers, tuned by measurement), push the post into followers' feed caches as before. Above it, store the post only on the celebrity's own timeline. At read time, assemble the feed from the precomputed cache plus a merge with the recent posts of the few celebrities this user follows. Normal posts arrive instantly at the cache; celebrity posts are pulled from one hot, well-replicated timeline only when someone actually reads.
A raw reverse-chronological list is the baseline; ranking layers on top. The signals are recency (newer beats older), affinity (you interact with this author often), and engagement predictions (a model guesses whether you will like, reply, or dwell on this post). Rank well and the product feels alive. Rank wrong — bury your best friend's post under a stranger's ad, or push outrage because outrage predicts clicks — and users churn, trust erodes, and regulators start reading your metrics. Ranking is a product decision wearing a model's clothes, and its failure modes are social, not just technical.
- Feed cache: per-user precomputed list of post IDs (the newest few hundred), filled by the fanout worker; this is what makes reads one lookup.
- Content cache: the post objects themselves — text, media URLs, counts — keyed by post ID, so hydrating a feed does not hit the database per post.
- Social-graph store: who follows whom — a sharded store answering “who are this author's followers?” on write and “which celebrities does this user follow?” on read.
Fanout is a question of when you pay. Push pays now, at write time, so every read is nearly free. Pull pays on every read, so writes are nearly free. Neither is wrong — the hybrid simply pays each way exactly where that way is cheap: push for the many small accounts where writes are cheap, pull for the few giant accounts where reads are rare and writes are catastrophic.
Why does the celebrity break push fanout, and what does the hybrid do with her posts? (Her 50M followers turn one post into 50M cache writes — minutes of write storm, most of it for followers who never read. The hybrid skips the push for her: her posts live on her own timeline and are merged into a reader's feed at read time only, from one hot, replicated source — so one write serves everyone who actually reads.)
Split the design into publishing and reading, and the rest follows: push fanout makes reads one lookup but melts on celebrities, pull fanout makes writes trivial but melts on every home page, and the hybrid — push for normal users, pull-and-merge for celebrities — is why real feeds work. Then feed cache, content cache, and the social graph carry the load, and ranking decides what users feel.
Pick any system you own where one write fans out to many readers. Count the fanout: average recipients per write, and the maximum. If the max is a hundred times the average, you have a celebrity problem — design the threshold where push becomes pull before your first famous customer finds it for you.
Product-Scale Systems