Product-Scale Systems
Design a chat system
A chat app feels trivial until you try to build one: a message must cross the internet in a fraction of a second, land on every device a user owns, and arrive in the right order — millions of times a day. Scope it tightly: 1:1 conversations and small groups, text only, no calls, no stories. That narrow scope still forces you to solve connections, presence, ordering, and offline sync. Four real problems, one lesson.
The naive design: every client asks “anything new for me?” every few seconds. Run the numbers. One million online users polling every 5 seconds is 200,000 requests per second — and the overwhelming majority come back empty, because people receive a message every few minutes, not every few seconds. You are burning nearly all your server capacity answering “no”. Worse, latency is built in: a message waits up to half the poll interval before anyone sees it. Polling is a fine prototype and a terrible product.
Long polling is the stepping stone: the client asks, and the server holds the request open until a message arrives or a timeout fires, then the client immediately re-asks. Empty responses disappear, but you still pay a full HTTP setup per message, and under load servers hold thousands of suspended requests. WebSocket is the real answer: upgrade the HTTP connection once, then keep one persistent, bidirectional channel per user. The server can now push a message the instant it exists — no asking, no waiting. One connection per user, held open for hours.
WebSocket solves delivery and creates a new problem: chat servers are stateful. A single machine can hold on the order of 50K–100K open connections, so a big service is a fleet of chat servers, each owning a fixed set of connections. You need a connection map — which user lives on which server — usually kept in a fast store like Redis. Rebalancing, server crashes, and deploys now mean reconnecting thousands of clients without a thundering herd. Connection management is not a detail; it is a subsystem.
Presence answers one question: who is online? The mechanism is a heartbeat — each client pings the presence service every few seconds, and a user with no heartbeat for ~30 seconds is marked offline. Status changes fan out only to that user’s friends or group members, never to everyone. Now the honest part: presence is eventually consistent by design. If a heartbeat is lost, a friend looks offline for half a minute while actually online. Nobody is harmed, so you accept the error and skip the expensive coordination that would make presence exact.
- Sender’s client pushes the message over its WebSocket to its chat server, which assigns the message an ID and a per-channel sequence number.
- The chat server drops the message onto a message queue — the write is now durable and the server can ack the sender without waiting for delivery.
- Workers consume the queue, persist the message in a key-value store keyed by channel ID, and look up each recipient’s chat server to push it down their connections — or to push notification if they are offline.
Why a sequence number per channel and not one global order? A global sequencer is a single process that must see every message in the system — a bottleneck and a single point of failure — and it buys you nothing, because messages in different channels have no meaningful order relative to each other anyway. Nobody cares whether a message in group A “came before” one in group B. Within one channel, though, order is the product: replies must follow the messages they answer. So each channel gets its own monotonically increasing counter, which is cheap to generate and trivially scales by shard.
The same sequence number solves multi-device sync. Every device remembers a cursor — the highest sequence it has seen per channel. On reconnect, the device sends its cursors and the server replays everything after them: your phone picks up exactly where your laptop left off, no duplicates, no gaps. If the user is fully offline, no connection exists to push to, so the send path hands the message to a push notification service instead. For group chat the fanout is per member: the message is written once to the channel, then pushed to every member’s connection — and with small groups, that multiplication is cheap.
Two questions. First: why is WebSocket preferred over polling for chat? (Polling at one request per 5 seconds per user means ~200K requests/s per million users, almost all empty, plus built-in latency; one persistent WebSocket per user lets the server push instantly with no wasted traffic.) Second: why is message ordering done per channel instead of globally? (A global sequencer is a bottleneck and single point of failure, and cross-channel order is meaningless — only within a channel does order carry semantics, so each channel keeps its own counter.)
A chat system is four decisions chained together: kill polling with persistent WebSocket connections and accept the statefulness that brings; make presence cheap and briefly wrong on purpose; write through a queue into a key-value store ordered by per-channel sequence numbers; and let those same cursors drive multi-device sync, push for the offline, and per-member fanout for groups. None of these is exotic — the craft is in saying why.
Open the network tab on a chat product you use (WhatsApp Web, Slack, Messenger) and watch what actually flows. Find the WebSocket connection, read a few frames, and notice the heartbeat messages. Then estimate for your own team: how many concurrent connections would your user base need, and how many chat servers at 50K connections each? Write the two numbers down — that is the seed of a real capacity plan.
Product-Scale Systems