Foundations of Scale
Data at scale: replicate, shard, distribute
Your database just became the bottleneck — again. Reads are drowning it, or the data no longer fits, or it is one hardware hiccup away from taking the company down with it. Three different diseases, three different cures.
In leader–follower replication, one leader accepts all writes and streams them to followers that serve reads. Reads scale horizontally — add a follower, get more read capacity. The trade-off is sync versus async: synchronous replication never loses a write but makes every write wait; asynchronous is fast but a crashed leader can take the newest writes with it. And when the leader dies, failover promotes a follower — which is exactly the moment your replication lag becomes visible to users.
With async replication, a write lands on the leader and reaches followers a few milliseconds — or a few seconds — later. So a user updates their profile, refreshes, gets routed to a lagging follower, and sees the old value: their own write has vanished. That is the read-your-writes problem. The subtler cousin is monotonic reads: two refreshes hit two different followers at different lag, and data appears to move backwards in time. The standard mitigations are cheap and boring: route a user's own reads to the leader for a short window after each write, have the client remember the timestamp or LSN of its last write and only read from a follower that has caught up to it, or pin a user to one follower so reads never go backwards. None of this makes replication synchronous — it just hides the lag from the people who would notice it.
When one database cannot hold the data or absorb the writes, split it: rows with shard key 1–1M go to server A, 1M–2M to server B. Choosing the shard key is the whole game — a bad key concentrates load instead of spreading it. The celebrity problem is the classic failure: one user's data is so hot that their shard melts while the others idle. And resharding later — moving rows between servers — is one of the most painful operations in engineering, so pick the key like you will live with it for years. You will.
Range-based sharding assigns contiguous key ranges to each shard — users A–F here, G–M there. The payoff is ordered scans: range queries and sorted iteration stay inside one shard and stay fast. The price is hotspots: shard by an auto-incrementing id or a timestamp, and every new write lands on the last shard while the rest idle. Hash-based sharding runs the key through a hash function, so writes spread almost perfectly evenly — no hotspot, ever. The price: ordering is destroyed, and 'give me users A through M' becomes a scatter-gather across every shard. Pick range when scans dominate and you can tolerate rebalancing; pick hash when even spread matters more than order.
Use SQL by default: transactions, joins, constraints, and forty years of hard-won reliability cover most applications comfortably. Reach for NoSQL when the scale or the access pattern genuinely demands it — massive write throughput, flexible schemas, simple key-value lookups at huge volume, or data that is naturally a document or a graph. The honest summary: SQL is the default, NoSQL is the exception you justify with numbers, not fashion.
How do you decide which server holds which key, without remapping everything when a server joins or leaves? Hash both servers and keys onto the same ring; each key belongs to the first server clockwise. Add or remove one server and only ~1/N of the keys move — everyone else stays put. This one idea quietly powers caches, sharded databases, and load balancers everywhere.
The naive reshard — hash key modulo N, then change N — remaps almost every row and takes the system down with it. The standard answer is a fixed number of logical shards, say 1,024, mapped onto far fewer physical nodes: rebalancing means moving whole logical shards between nodes, a routing-table change instead of a data-model surgery. During the migration itself you double-write — every write goes to both the old and the new location — while a background job backfills, and reads cut over only after the copy verifies. And the rule that saves you years later: pick many more logical shards than nodes on day one, so growing the fleet is always 'hand some shards to the new machine' and never 'rethink the partition scheme under load.'
Replication buys you reads and survival; sharding buys you writes and size. They solve different problems — and most real systems at scale end up needing both, replication inside each shard.
Your app does 95% reads and its single database is dying — replication or sharding? And which one helps when the data simply stops fitting on one disk? (Replication for the read flood — followers absorb reads; sharding for the size problem — no replica makes the data smaller.)
Copy the data to survive and to read more; split the data to write more and to hold more. Default to SQL, justify NoSQL with numbers, and use consistent hashing whenever keys must spread across servers that come and go.
Take the largest table in a system you know. Write down: what would you shard it by, which rows would be the “celebrity,” and what breaks in your queries the moment the data lives on two machines. If the answers scare you, you just learned why sharding is the last resort.
Foundations of Scale