// Product-Scale Systems

One event, millions of notifications

System Design track · 20 min

By the end of this lesson you can
  • Draw the path from one event to push, SMS and email, with a queue and a worker pool for each channel.
  • Make retries safe with exponential backoff, a dead-letter queue and an idempotency key.
  • Work out how long a big send will take when a provider caps your rate.

A user posts one comment. A million followers should hear about it, by push, SMS or email, within seconds. The person who commented should not wait, and your servers should not catch fire.

This is the notification system. It is where the ideas from earlier lessons, queues, rate limits and retries, stop being theory and start paying rent.

The path of one notification
  1. Something happens: a comment, an order, a price drop. The service where it happened publishes an event and moves on. It never waits for a million phones.
  2. The notification service works out who should be told, on which channels, with what text, after checking each user's settings. Then it puts one job per message into a queue for each channel: push, SMS, email.
  3. Workers pull jobs from each queue and call the outside providers: APNs (Apple) and FCM (Google) for push, an SMS gateway for texts, an email service over SMTP for email.
  4. A failed send is retried later. A message that keeps failing goes to a dead-letter queue, where a person or a script can look at it instead of it vanishing.
The queue is the hero

Think of a restaurant kitchen. The waiter writes the order on a ticket, pins it on the rail, and goes back to the tables. He does not stand at the stove until the food is ready. The queue (طابور الرسائل) is that ticket rail.

It splits the event from the delivery: the comment saves in 50 ms, and the notifications go out over the next minute. It absorbs spikes: when a celebrity posts, ten million jobs wait in line instead of crushing your workers. And it makes retries natural, because a failed send is just a job put back in line. Remove the queue, and every spike becomes an outage.

One worker pool per channel, sized by the provider

Push, SMS and email get separate queues and separate worker pools, because they behave very differently. A push through APNs is nearly free and fast. An SMS costs real money, and a gateway may accept only 100 per second from your sender. Email is slower still.

Size each pool for its channel's volume, but also for its provider's limit. APNs, FCM and SMS gateways all limit how fast you can send. Go past the limit and messages fail, or worse, your sender account gets slowed down or banned.

So put a token bucket, from the rate limiter lesson, in front of each provider. Let the provider's limit set the pace, not your ambition.

Retry carefully, and never send twice

A failed send is retried with exponential backoff: wait 1 s, then 2 s, then 4 s, then 8 s, plus a little random jitter. The jitter matters. Without it, thousands of workers that failed together all retry at the same second and knock the provider over again.

After a fixed number of attempts, the job moves to the dead-letter queue to wait for inspection, instead of blocking the line or disappearing.

There is one more catch. Queues deliver at least once, so the same job can arrive twice even when nothing failed. The fix is an idempotency key: event ID + recipient + channel, recorded before sending. A worker that sees a key it already sent simply drops the job. Without it, one order can text a customer twice.

Not all messages are equal

A one-time password (OTP) must arrive in seconds. A marketing message can wait a few minutes. So give OTPs and security alerts their own high-priority queue that workers empty first, and let promotions use whatever capacity is left.

And skip work entirely where you can. Check opt-outs, quiet hours and channel settings before a job enters the queue, not at the worker. Every message filtered out early is capacity you never spend and a user you never annoy. A message someone asked not to get is worse than no message.

Triggera system eventNotification svcchecks prefs / opt-outPush queuePush workersAPNs / FCM→ user devicesSMS queueSMS workersSMS gateway→ user devicesEmail queueEmail workersSMTP→ user devicesRetry → dead-letterfail
Read it left to right: event, settings check, one queue per channel (with a fast lane for OTPs), worker pools held to each provider's limit, then APNs, FCM, SMS and email. Failures loop through retries and end in the dead-letter queue.
A flash sale, by the numbers

Marketing wants to text 2 million customers about a flash sale. Your SMS gateway allows 100 messages per second.

2,000,000 ÷ 100 = 20,000 seconds. Divide by 3,600 and you get about 5.5 hours. The last customer hears about the sale five and a half hours after the first.

That is a business conversation, not a bug. Either start earlier, buy more sending capacity from the provider, or move most of the campaign to push, which is far cheaper and faster. Doing this math before launch day saves a very awkward meeting after it.

Test yourself

Your SMS gateway allows 100 messages per second, a flash sale queues 2 million texts, and the queue delivers every job at least once. What controls the speed, what happens to failures, and how do you stop duplicates?

Answer in your head first, then check.

Takeaway

A notification system is really a buffering problem. Queues per channel absorb the spike. Worker pools empty them at exactly the pace each provider allows. Backoff and a dead-letter queue make failure survivable. Idempotency keys make retries safe. And a priority lane makes sure the OTP never waits behind the newsletter.

Do it this week

On one page, sketch the notification system for a product you know: the event, the settings check, one queue per channel with a priority lane, worker pools labelled with each provider's rate limit, and the retry path ending in a dead-letter queue. Then write one sentence: where exactly is a duplicate send stopped? One page, one hour, no tools.

What you can do now
  • Draw the path from one event to push, SMS and email, with a queue and a worker pool for each channel.
  • Make retries safe with exponential backoff, a dead-letter queue and an idempotency key.
  • Work out how long a big send will take when a provider caps your rate.

Product-Scale Systems