Foundations of Scale
From one server to millions
Your side project just hit the front page. Traffic doubles every hour — and everything runs on one machine under your desk. Tonight you will learn, the hard way, what every big system already knows.
One machine has hard limits: finite CPU, finite memory, one network card. Worse, it is a single point of failure — if it dies, your entire product dies with it. Scaling is the art of removing these limits one bottleneck at a time.
- Separate the database onto its own machine. Your app server and your database stop fighting each other for CPU and memory — each can now be tuned for its own job.
- Scale vertically first: buy a bigger machine. Zero code changes, instant relief. But the ceiling is real — there is always a biggest machine, and it is always expensive.
- Scale horizontally: add more web servers behind a load balancer. The balancer spreads requests across the pool, and any server can die without taking the site down.
- Replicate the database: one leader takes writes, followers take reads. Most apps read far more than they write, so read replicas multiply your capacity where it hurts.
- Add a cache. Expensive queries and hot data get served from memory in microseconds, and the database finally gets to breathe.
- Push static assets to a CDN. Images, videos, and files get served from edge servers near your users instead of crossing the planet from your data center.
- Make the web tier stateless — move sessions out of server memory into a shared store. Now any server can handle any request, and scaling out is just adding boxes.
- Shard the database. When one database can't hold the data or the writes anymore, split it by a shard key across many databases. This is the last resort, not the first.
Put ballparks on the staircase. One tuned web server serves thousands to tens of thousands of requests per second — say 10K req/s for a simple JSON endpoint. A single well-provisioned relational database takes a few thousand queries per second before the disk and the locks push back — call it 3–5K. An in-memory cache does 100K+ operations per second on one node, because memory has no seek time and no write-ahead log. So at 2K req/s, one server and one database are fine. At 20K req/s you need three web boxes behind a balancer, the database is sweating, and a cache absorbing 90% of reads turns a 20K query storm into 2K the database can actually carry.
Step seven is where teams cheat. The lazy fix is sticky sessions: pin each user to one server at the load balancer. It works until a pinned server dies — then every user glued to it loses their session at once, and failover becomes a mass logout. Worse, load stops being balanced: a few heavy users stick to one box and it melts while its neighbors idle. The real fix is moving session state out of the servers entirely, into a shared cache or central session store that any web server can read. Now a dead server is a non-event — the balancer just routes to the survivors.
Every scaling move is the same loop: find the bottleneck → add a layer → accept the new complexity that layer brings. Scale is a staircase, not a switch — you climb one step when the current step starts to crack, never five steps at once.
The classic mistake is distributing before you need it. Every layer you add — load balancers, replicas, caches, shards — is complexity you pay for every single day: in bugs, in deploys, in on-call nights. A boring single server that handles your load beats a distributed system you don't need.
One layer is missing from this staircase: the message queue. It enters the moment work does not have to happen inside the request — sending email, resizing images, charging cards — or when a spike would flatten your database: writes land in the queue instantly, and workers drain it at a pace the database can survive. That is a different kind of scaling: not serving more requests, but decoupling when work happens from when it is requested. Hold that thought — it earns its own lesson.
Your web servers keep user sessions in memory, and you add a second server behind a load balancer — what breaks? (The balancer sends a user's next request to the other server, which has never seen their session: they are suddenly logged out. The fix is a stateless web tier — sessions move to a shared store like a cache.)
Nobody designs for a million users on day one. They start with one server, watch it crack, and climb the staircase: separate, scale up, scale out, replicate, cache, CDN, go stateless, shard — one bottleneck at a time.
Draw your current system as boxes on one page: users, servers, database, storage. Then circle every single point of failure — every box whose death takes the whole product down. That page is your scaling roadmap.
Foundations of Scale