// Product-Scale Systems

Design a chat system

System Design track · 25 min

By the end of this lesson you can
  • Explain with numbers why chat uses WebSocket instead of polling, and what that costs you.
  • Trace a message from sender to every device of the receiver, including when they are offline.
  • Use per-chat sequence numbers to keep order and sync many devices.

A chat app looks easy until you build one. A message must cross the internet in a fraction of a second, reach every device the person owns, and arrive in the right order, millions of times a day.

Keep the scope tight: one-to-one chats and small groups, text only, no calls, no stories. Even that small scope forces you to solve four real problems: connections, presence, ordering and offline sync.

Why polling fails

The naive design: every app asks the server “anything new for me?” every few seconds. This is called polling.

Do the math. One million online users asking every 5 seconds is 1,000,000 ÷ 5 = 200,000 requests per second. Almost all of them come back empty, because people get a message every few minutes, not every few seconds. You are spending most of your servers on saying “no”.

It is also slow by design. A message waits, on average, half the polling interval before anyone sees it. Polling is fine for a prototype and bad for a product.

Long polling, then WebSocket

Long polling is the middle step. The app asks, and the server keeps the request open until a message arrives or a timeout passes. Then the app asks again right away. Empty answers disappear, but each message still costs a full HTTP request, and under load the servers hold thousands of waiting requests.

WebSocket is the real answer. The app opens one connection, upgrades it once, and keeps it open for hours. The channel works in both directions, so the server can push a message the instant it exists. No asking, no waiting. It is like keeping the phone line open instead of calling every five seconds to ask whether there is news.

The price: servers that remember

WebSocket solves delivery and creates a new problem: chat servers now hold state. One machine can keep roughly 50,000 to 100,000 connections open, so a large service runs a fleet of chat servers, each holding its own set of users.

You need a connection map, which user is connected to which server, usually kept in a fast store like Redis. Restarts, crashes and deploys now mean reconnecting thousands of users at once without all of them stampeding back in the same second. Managing connections is a whole subsystem, not a detail.

Client 1Client 2offlineChat serversWebSocketpersistentMessage queueKV message storeper-channel sequence numbers (channel seq #)Presence serviceheartbeatsheartbeatPush serviceAPNs / FCMfor offline users
Apps hold WebSocket connections to a fleet of chat servers. A sent message goes chat server, then queue, then key-value store. Presence and push notifications sit on the side.
Presence: allowed to be a little wrong

Presence answers one question: who is online? Each app sends a small heartbeat to the presence service every few seconds. If a user sends no heartbeat for about 30 seconds, they are marked offline. Status changes go only to that user's friends or group members, never to everyone.

Now the honest part. Presence is eventually consistent on purpose. If one heartbeat is lost, a friend may look offline for half a minute while actually online. Nobody is harmed, so you accept that small error and skip the expensive coordination that perfect accuracy would need.

How a message is sent
  1. The sender's app sends the message over its WebSocket to its chat server. The server gives the message an ID and a sequence number for that chat.
  2. The chat server puts the message on a queue. Now it is safely stored, so the server can confirm to the sender (the first tick) without waiting for delivery.
  3. Workers read from the queue, save the message in a key-value store keyed by chat ID, then look up each receiver's chat server in the connection map and push the message down their connection. If the receiver is offline, a push notification goes out instead.
Order inside each chat, not across the whole system

Why a sequence number per chat, and not one global order for all messages? A global counter is one process that must see every message in the system. That makes it a bottleneck and a single point of failure. And it buys nothing, because no one cares whether a message in group A came “before” one in group B.

Inside one chat, though, order is everything. A reply must come after the message it answers. So each chat gets its own sequence number, a counter that only goes up. It is cheap to generate and it scales naturally when you shard chats across servers.

Phone and laptop, in sync

The same sequence number solves sync across devices. Each device remembers a cursor: the highest sequence number it has seen in each chat.

Say your laptop has seen messages up to #41 in the family group, and your phone was off and has only #37. When the phone reconnects, it sends “family group: 37”. The server replies with #38 to #41. Nothing is missing and nothing is repeated.

If the user is fully offline, there is no connection to push to, so the send path hands the message to the push notification service. In a group, the message is stored once for the chat, then pushed to each member's connection. With small groups, that copying is cheap.

Test yourself

Two questions. Why does chat use WebSocket rather than polling? And why are messages ordered per chat instead of globally?

Answer in your head first, then check.

Takeaway

A chat system is four decisions in a row. Replace polling with open WebSocket connections, and accept the state that comes with them. Keep presence cheap and slightly wrong on purpose. Send through a queue into a key-value store, ordered by a sequence number per chat. And let those same numbers drive sync across devices, push for offline users, and delivery to group members.

Do it this week

Open the browser's network tab on a chat product you use (WhatsApp Web, Slack, Messenger). Find the WebSocket connection, read a few frames, and spot the heartbeats. Then estimate for your own product: how many users are online at the same time at peak, and how many chat servers would you need at 50,000 connections each? Write down both numbers. That is the start of a real capacity plan.

What you can do now
  • Explain with numbers why chat uses WebSocket instead of polling, and what that costs you.
  • Trace a message from sender to every device of the receiver, including when they are offline.
  • Use per-chat sequence numbers to keep order and sync many devices.

Product-Scale Systems