// Foundations of Scale
From one server to millions
System Design track · 20 min
- List the scaling moves in order, from one server to a sharded database, and say which problem each one removes.
- Find the single points of failure in a simple system diagram.
- Explain why web servers must be stateless before you add more of them.
You launch a small online sweets shop. Everything runs on one server: the website, the database, the product photos. Two days before Eid, a popular influencer shares your link. Orders double every hour. Pages get slower, then the site stops answering at all.
Every big company you know has lived through this night. The good news is that the way out is well known, and its steps come in a clear order.
One server has hard limits: a fixed amount of CPU, a fixed amount of memory, one network card. When any of these fills up, everything slows down together.
The bigger danger is that it is a single point of failure: one part that, if it breaks, takes the whole product down with it. A dead disk or a bad update, and your shop disappears from the internet.
Every scaling move follows the same loop. Find the bottleneck, the part that chokes first. Add a layer that relieves it. Then accept the new complexity that layer brings with it.
That is why scaling is a staircase, not a switch. You climb one step when the step you are standing on starts to crack. You never jump five steps at once.
- Move the database to its own machine. The app and the database stop fighting over the same CPU and memory, and you can tune each one for its own job.
- Scale up first (vertical scaling): buy a bigger machine with more memory and a stronger CPU. No code changes, instant relief. But there is a ceiling. There is always a biggest machine, and it is always expensive.
- Scale out (horizontal scaling): add more web servers and put a load balancer in front of them. Think of the person at a bank entrance who sends each customer to the next free counter. Now any one server can die and the site stays up.
- Replicate the database. One leader copy takes every write; follower copies answer reads. Most apps read far more than they write, so followers add capacity exactly where it hurts.
- Add a cache: fast memory that keeps the answers to repeated questions. Popular data comes back in under a millisecond, and the database finally gets room to breathe.
- Put static files on a CDN, a network of servers spread across many cities. Images and videos reach each user from a nearby server instead of crossing continents from your data centre.
- Make the web servers stateless: move user sessions out of each server's memory into a shared store. Now any server can handle any request, and scaling out is just adding machines.
- Shard the database when one database can no longer hold the data or keep up with the writes: split the rows across several databases by a key, such as the user ID. This is the last step you take, not the first.
Put ballpark numbers on the staircase. One well-tuned web server handles around 10,000 requests per second for a simple API endpoint that returns JSON. (An API is the doorway an app uses to ask for data.) One strong relational database handles a few thousand queries per second, say 3,000 to 5,000, before the disk and the locks start pushing back. An in-memory cache does more than 100,000 operations per second on a single machine.
Now do the maths, assuming each request needs one database query. At 2,000 requests per second, one server and one database are fine. At 20,000 requests per second you want three web servers behind a load balancer: two to carry the load, one spare. The database cannot take 20,000 queries. But if the cache answers 90% of reads, only 10% reach it: 20,000 × 10% = 2,000 queries per second, which it can carry.
Now try it yourself. Drag the user count up slowly, watch which part chokes first, and see which layer appears to fix it.
Step seven is where teams take a shortcut. The easy fix is called sticky sessions: you tell the load balancer to always send each user to the same server.
It works until that server dies. Then every user tied to it is logged out at once. Load also stops being balanced: a few heavy users stick to one machine and it overheats while its neighbours sit idle.
The real fix is to take session data out of the servers completely and keep it in a shared cache or session store that any server can read. After that, a dead server is a non-event. The load balancer sends traffic to the others and nobody notices.
The classic mistake is building a distributed system before you need one. Every layer you add, whether a load balancer, replicas, a cache or shards, is complexity you pay for every day: in bugs, in deployments, in nights on call.
A boring single server that carries your load beats a clever distributed system you don't need yet.
One layer is missing from this staircase: the message queue. A queue is a waiting line for work that does not have to happen inside the request, like sending an email, resizing a photo or charging a card. The request drops the job in the queue and returns at once, and background workers pick jobs up at a pace the database can survive.
That is a different kind of scaling. It does not serve more requests; it separates when work is asked for from when it is done. Keep this idea close, you will need it often.
Your web server keeps user sessions in its own memory. You add a second server behind a load balancer. What breaks, and how do you fix it?
Answer in your head first, then check.
Nobody designs for a million users on day one. You start with one server, watch it crack, and climb: separate the database, scale up, scale out, replicate, cache, CDN, go stateless, shard. One bottleneck at a time.
Draw your current system as boxes on one page: users, servers, database, file storage. Then circle every single point of failure, every box whose death takes the whole product down. That page is your scaling plan.
- List the scaling moves in order, from one server to a sharded database, and say which problem each one removes.
- Find the single points of failure in a simple system diagram.
- Explain why web servers must be stateless before you add more of them.
Foundations of Scale