Product-Scale Systems
One trigger, millions of deliveries
A user posts one comment. A million followers need to know — by push, SMS, and email — within seconds, without the commenter waiting, and without the servers catching fire. This is the notification system, and it is where queues, rate limits, and retries stop being theory.
- An event happens — a comment, an order, a price drop. The triggering service publishes it and moves on; it never waits for a million phones to answer.
- The notification service enriches the event: who should be told, on which channels, with what text — after checking their settings. Then it fans out to one message queue per channel: push, SMS, email.
- Workers pull from each queue and call the third-party providers: APNs and FCM for push, an SMS gateway for texts, SMTP for email.
- Failures retry with backoff; messages that keep failing land in a dead-letter queue where a human or a script can inspect them instead of losing them silently.
The queue decouples the trigger from the delivery: the comment saves in 50ms while the notifications drain over the next minute. It buffers spikes — a celebrity posts and ten million jobs wait patiently in line instead of melting your workers. And it makes retries natural, because a failed send is just a message put back. Remove the queue and every spike becomes an outage.
Push, SMS, and email get separate queues and separate worker pools, because their shapes differ wildly: push to APNs costs fractions of a cent and moves fast, an SMS costs real money per message and a gateway might accept only 100 per second per sender, email crawls through SMTP. Size each pool to its channel's volume — but also to its provider's ceiling. APNs, FCM, and every SMS gateway rate-limit you, and blowing past the limit does not just fail messages; it can get your sender account throttled or banned outright. So put a token bucket per provider in front of each pool and let the provider's limit, not your ambition, set the drain rate.
A failed send retries with exponential backoff — 1s, 2s, 4s, 8s — plus jitter, so a provider hiccup does not meet a synchronized thundering herd of retries. After a fixed number of attempts the message moves to a dead-letter queue, where it waits for inspection instead of blocking the line or vanishing. But retries collide with a fact of life: queues deliver at least once, so the same job can legitimately arrive twice even without a failure. The fix is an idempotency key — the event ID plus the recipient and channel — recorded before the send; a worker that sees a key it already sent simply drops the duplicate. Without it, every retry risks texting a user twice for one order.
A one-time password must land in seconds; a marketing blast can wait minutes. So route by priority: give OTPs and security alerts their own high-priority queue whose workers drain first, and let promotional jobs fill the leftover capacity. And save yourself the work entirely where you can: check opt-outs, quiet hours, and channel settings before a job is ever enqueued, not at the worker. Every notification filtered out upstream is capacity you never spend and a user you never annoy — a message someone asked not to receive is worse than none at all.
Your SMS gateway allows 100 messages per second, a flash sale queues 2M texts, and the queue delivers every job at least once. Walk the design: what shapes throughput, retries, and duplicates? (A token bucket throttles the SMS worker pool to 100/s, so the batch drains over ~5.5 hours — tell the business or buy more sender capacity. Failures retry with exponential backoff plus jitter, capped at a fixed attempt count, then land in the dead-letter queue. And because at-least-once delivery plus retries can repeat a send, each job carries an idempotency key — event + recipient + channel — checked before calling the gateway, so no user gets the same text twice.)
A notification system is a buffering problem wearing a delivery costume: queues per channel absorb the spike, worker pools drain them at exactly the rate each provider tolerates, backoff and dead-letter queues make failure survivable, idempotency keys make retries safe, and priority lanes make sure the OTP never waits behind the newsletter.
Sketch the notification system for a product you know on one page: the trigger, the enrichment step with its settings check, one queue per channel with a priority lane, worker pools annotated with each provider's rate limit, and the retry path ending in a dead-letter queue. Then answer in one sentence where a duplicate send is prevented. One page, one hour, no tools.
Product-Scale Systems