Design a Notification System

A notification system looks trivial until you remember that one product event — a new like, a shipment update, a fraud alert — may need to reach tens of millions of users within seconds, across four different channels (push, SMS, email, in-app), through external providers you do not control, while respecting each user's opt-outs and quiet hours, never sending the same message twice, and surviving the inevitable APNs or FCM hiccup. The hard parts are not the "send an email" call; they are the fan-out from one event to millions of deliveries, the at-least-once-plus-idempotent delivery semantics, the third-party failure handling, and the preference and rate-limiting logic that keeps you from spamming users. Get those wrong and you either drop critical alerts or wake up your entire user base at 3 a.m. with a retry storm.

Step 1 — Clarify Requirements

A notification system is a classic interview prompt because the naive answer ("call SendGrid in a loop") collapses the moment you introduce scale, multiple channels, and reliability. Pin down scope first.

Functional requirements

• Send notifications across four channels: mobile push (iOS/Android), SMS, email, and in-app (a feed/inbox the client polls or receives over a websocket).
• Support triggered (transactional, event-driven: OTP, order shipped) and scheduled/broadcast (marketing campaigns to a segment) notifications.
• Respect user preferences: per-category channel opt-in/out, quiet hours, frequency caps, and global unsubscribe.
• Render content from templates with per-locale variants and runtime variables.
• Provide delivery status: queued, sent, delivered, failed, opened (where the provider reports it).

Non-functional requirements

• Scale — handle bursty fan-out: a single broadcast may produce 50M+ deliveries; steady state is millions of transactional sends per hour.
• Low trigger latency — the API that accepts a notification request returns in <100 ms; actual delivery is async.
• Reliability — critical notifications (OTP, security) must not be silently dropped; target at-least-once delivery with idempotency.
• No duplicates — a user should never receive the same logical notification twice, even when the pipeline retries.
• Extensibility — adding a new channel (e.g., WhatsApp) should not require rewriting the core.

Step 2 — Capacity Estimation

Sizing tells you where the bottleneck is — it is almost never the database; it is the queue throughput and the provider rate limits.

Traffic

• Assume 100M DAU and an average of 5 notifications/user/day → 500M notifications/day ≈ ~5,800/sec average.
• Peak is spiky: a marketing broadcast to 50M users fires in minutes. If we want it drained in 10 minutes, that is 50M / 600 s ≈ ~83,000 deliveries/sec sustained — the design must absorb 10–20× the average as a burst.
• Channel mix is uneven: push dominates (≈70%), in-app (≈20%), email (≈8%), SMS (≈2%, but the most expensive and rate-limited).

Storage

• Notification log record: ~300 bytes (ids, channel, status, timestamps, template_id). 500M/day × 300 B ≈ ~150 GB/day — this goes to a horizontally scalable store with TTL (e.g., 30–90 days), not a single SQL box.
• Device tokens: 100M users × ~2 devices × ~100 B ≈ ~20 GB — small, hot, kept in a fast store and cache.
• Preferences: 100M users × ~200 B ≈ ~20 GB — read on every fan-out, so cached aggressively in Redis.

Step 3 — API Design

The public surface is small. The producer describes what to send and to whom; the system decides how and when.

# Send a notification (transactional)
POST /api/v1/notifications
     Authorization: Bearer <service-token>
     Idempotency-Key: "order-9182-shipped"   // dedup at the edge
     { "user_id": "u_123",
       "template_id": "order_shipped_v3",
       "channels": ["push", "email"],   // optional; else use prefs
       "variables": { "order_id": "9182", "eta": "Jun 30" },
       "priority": "transactional" }
     → 202 { "notification_id": "ntf_8f2a", "status": "accepted" }

# Broadcast to a segment (async, returns a job handle)
POST /api/v1/broadcasts
     { "segment_id": "seg_us_active", "template_id": "promo_summer",
       "priority": "promotional", "send_at": "2026-07-01T16:00:00Z" }
     → 202 { "broadcast_id": "bc_55", "status": "scheduled" }

# Query delivery status
GET /api/v1/notifications/{id}
     → { "status": "delivered", "channel": "push",
         "attempts": 1, "delivered_at": "..." }

# Update user preferences
PUT /api/v1/users/{id}/preferences
     { "marketing": { "push": false, "email": true },
       "quiet_hours": { "start": "22:00", "end": "08:00", "tz": "America/New_York" } }

Two design points. First, the Idempotency-Key header lets the caller make the trigger safe to retry — if the order service crashes and re-POSTs "order-9182-shipped", the API returns the same notification_id instead of sending twice. Second, the endpoint returns 202 Accepted, not 200: delivery is asynchronous, so the API's only job is to validate, dedup, persist the request, and enqueue. Everything expensive happens downstream.

Step 4 — High-Level Architecture

The system is a pipeline of decoupled stages connected by queues. The key principle: the trigger path is cheap and fast; the delivery path is parallel and absorbs failure.

Producers (order svc, social svc, risk svc, campaign tool)
   │  POST /notifications  |  POST /broadcasts
   ▼
[ Notification API ] ── validate, edge-dedup (Idempotency-Key), persist
   │  publish 1 event
   ▼
[ Kafka: "notification-events" ]   ← absorbs the burst
   │
   ▼
[ Fan-out / Preference workers ]
   ├── expand audience (1 event → N users)
   ├── load preferences + device tokens (Redis)
   ├── drop opted-out / quiet-hours / over-cap
   └── emit 1 job per (user, channel)
   ▼
[ Per-channel queues ]   push | sms | email | in-app
   │             │         │        │
   ▼             ▼         ▼        ▼
[ Push wkr ] [ SMS wkr ] [ Email ] [ In-app wkr ]
   │  APNs/FCM   │ Twilio   │ SES     │ writes to inbox + websocket push
   ▼             ▼          ▼         ▼
[ Provider adapters ] ── retry/backoff, rate-limit, DLQ on exhaustion
   │  delivery receipts / webhooks
   ▼
[ Status store + Observability ]  (per-notification trace, funnel metrics)

Why a queue between every stage? Because each stage has a different and independent failure mode and throughput ceiling. The fan-out worker can expand an audience far faster than Twilio will accept SMS; the message queue in front of each channel is the shock absorber that lets the fast stages run flat-out while the slow, rate-limited providers drain at their own pace. Kafka also gives you replayability: if the email worker has a bug, you fix it and reprocess from the offset without losing events.

Step 5 — Channel Adapters: APNs, FCM, SMS, Email, In-App

Each channel is fundamentally different. The shared abstraction is a channel adapter interface — send(job) → {delivered | retryable_error | permanent_error} — but the implementations diverge sharply.

Push: APNs and FCM specifics

Push is the highest-volume channel and has the most operational subtlety. APNs uses a long-lived HTTP/2 connection — you authenticate once with a provider JWT (valid ~1 hour, refreshed) and multiplex thousands of sends over the same connection rather than reconnecting per message. FCM lets you batch up to 500 messages per call. Both return a per-token result: a success, a retryable error (provider 5xx, connection reset), or a permanent error (410 Gone / BadDeviceToken / UNREGISTERED). Permanent errors are a token-management signal, not a retry signal — the worker emits a "prune this token" event so a future send never wastes a slot on a dead device.

In-app: online vs offline

In-app is the one channel you fully control. The worker writes the notification to the user's inbox table (so it survives offline), then — if the user has a live websocket connection — pushes it in real time. If they are offline, the inbox row is enough; the client fetches unread items on next open. This dual write (durable inbox + best-effort live push) is what makes in-app feel instant without losing notifications for offline users.

Step 6 — The Fan-out Problem

Fan-out is the heart of the system: turning one trigger event into N per-(user, channel) delivery jobs. Two flavors, with very different cost profiles.

Triggered fan-out (1 → few)

A transactional event targets one user (or a small set). Fan-out is cheap: load the user's preferences and device tokens, decide channels, emit a handful of jobs. This path is latency-sensitive and runs on a dedicated high-priority lane.

Broadcast fan-out (1 → millions)

A campaign targets a segment of 50M users. Doing this in one worker is impossible — it would run for hours and any crash loses progress. The pattern is hierarchical fan-out with checkpointing:

• The broadcast is split into chunks (e.g., 10K users each) by paging the segment with a cursor. Each chunk becomes its own message.
• Many fan-out workers consume chunks in parallel; each expands its 10K users into per-channel jobs.
• Progress is checkpointed per chunk, so a worker crash only re-does one 10K chunk, not the whole 50M broadcast — and because consumers are idempotent, re-doing a chunk does not double-send.

This is structurally the same fan-out-on-write vs fan-out-on-read tension you see in a news feed: precompute deliveries up front (write amplification, fast at send) versus compute at read time. For notifications the answer is almost always fan-out-on-write — a notification only matters if it is pushed, so there is no "read time" to defer to.

Step 7 — Template Service and Personalization

Producers must not send raw strings. If the order service hard-codes "Your order shipped!", you cannot localize it, A/B test it, or fix a typo without a deploy across every producer. A template service centralizes content.

• Versioned templates — order_shipped_v3 is an immutable, reviewed artifact. Producers reference a template ID plus a variable map; rendering happens at send time.
• Per-locale variants — each template has localized bodies; the renderer picks based on the user's locale, falling back to a default.
• Per-channel rendering — the same logical notification renders differently per channel: a 40-char push title, a full HTML email, a 160-char SMS. The template bundles all channel renditions.
• Safe interpolation — variables are escaped per channel (HTML-escape for email, no injection for SMS) to prevent content-injection bugs.

id: order_shipped_v3
category: transactional
locales:
  en:
    push:  { title: "Shipped!", body: "Order {{order_id}} arrives {{eta}}" }
    email: { subject: "Your order is on the way", html: "templates/shipped.html" }
    sms:   { body: "Order {{order_id}} ships, ETA {{eta}}. Track: {{url}}" }
  es:
    push:  { title: "¡Enviado!", body: "Pedido {{order_id}} llega {{eta}}" }

Step 8 — User Preferences and Opt-out

Preferences are checked in the fan-out path, before a job is enqueued — never send first and filter later. The preference service answers, for a given (user, category, channel): is this allowed right now?

• Per-category, per-channel opt-in — a user might allow security emails but disable marketing push. Stored as a matrix of (category × channel) booleans.
• Quiet hours — no non-critical notifications during the user's sleep window, evaluated in their timezone. A notification generated at 03:00 local is either dropped (promotional) or deferred to the morning (digest).
• Frequency caps — "at most 3 marketing pushes per day." Enforced with a Redis counter per (user, category, day) that the fan-out worker increments.
• Global unsubscribe + legal — an unsubscribed user is hard-filtered. Email must honor one-click unsubscribe (RFC 8058); SMS must honor STOP.

Step 9 — Rate Limiting and Deduplication

Two distinct concerns that both protect the user and the providers.

Rate limiting

There are three levels, and a good answer names all three. Per-user frequency caps (product-level: don't annoy the user). Per-provider limits (operational: APNs/Twilio/SES will throttle or ban you if you exceed their quotas) enforced with a token bucket in front of each adapter so the worker pool self-throttles to the provider's ceiling. Per-tenant limits if you are a platform serving multiple internal teams, so one team's runaway campaign cannot starve another. This is the same machinery covered in depth in designing a rate limiter — a distributed token bucket or sliding window backed by Redis.

Deduplication

Dedup is what makes at-least-once delivery tolerable. The same logical notification can arrive at the consumer multiple times — the producer retried the trigger, Kafka redelivered after a rebalance, or a broadcast chunk was reprocessed after a crash. The consumer must collapse these to a single send:

def deliver(job):
    # dedup key is deterministic across retries
    key = f"dedup:{job.notification_id}:{job.user_id}:{job.channel}"
    # SETNX returns False if the key already exists → already sent
    first_time = redis.set(key, 1, nx=True, ex=86400)
    if not first_time:
        metrics.incr("dedup.suppressed")   # dropped a duplicate
        return
    result = adapter.send(job)        # call APNs / Twilio / SES
    if result.permanent_error:
        redis.delete(key)             # let a corrected retry through
        prune_token(job)

The dedup key is notification_id + user_id + channel so the same notification on push and email both go out, but a redelivered push job is suppressed. The TTL (24 h) bounds memory while covering any realistic retry window. The subtle bug to avoid: setting the dedup key after a successful send means a crash between send and set re-sends; setting it before and never clearing it on a permanent error means a fixable failure is wrongly suppressed forever. The ordering above — claim the key first, release it only on permanent error — handles both.

Step 10 — Retries, DLQ, and Delivery Guarantees

Providers fail constantly at scale: transient 5xx, connection resets, 429 throttling. The retry policy must distinguish failure classes and never retry forever.

Exponential backoff with jitter (1s, 2s, 4s… plus randomization) prevents a provider blip from turning into a synchronized retry storm where every worker retries in lockstep and re-overwhelms the recovering provider. After the attempt cap, the job moves to a dead-letter queue — a separate topic where exhausted jobs land for inspection, alerting, and optional manual or automated replay once the root cause is fixed. The DLQ is what turns "we silently dropped 200K OTPs during the APNs outage" into "200K jobs are parked in the DLQ and replayed automatically when APNs recovered."

Why at-least-once, not exactly-once

Exactly-once delivery to an external provider is unachievable: after you call Twilio and it sends the SMS, the acknowledgement back to you can be lost. You cannot tell "sent but ack lost" from "not sent," so a safe system retries — which means the SMS could go out twice. The honest, standard answer is at-least-once on the wire plus idempotent consumers: you accept that the pipeline may attempt a send more than once, and you make duplicate attempts harmless via the dedup key from Step 9. This collapses to effectively-once from the user's perspective without claiming a guarantee the providers cannot give you.

Step 11 — Data Model

Three stores, each chosen for its access pattern: device tokens and preferences are hot key-value reads; the notification log is a high-volume append.

-- Device tokens (hot KV; cache in Redis)
CREATE TABLE device_tokens (
  user_id    BIGINT,
  platform   VARCHAR(10),       -- ios | android | web
  token      VARCHAR(255),
  last_seen  TIMESTAMP,
  PRIMARY KEY (user_id, platform, token)
);

-- Preferences (read on every fan-out; cache aggressively)
CREATE TABLE preferences (
  user_id    BIGINT      PRIMARY KEY,
  matrix     JSONB,                 -- {category: {channel: bool}}
  quiet_start TIME, quiet_end TIME, tz VARCHAR(40),
  unsubscribed BOOLEAN  DEFAULT FALSE
);

-- Notification log (high-volume append; NoSQL / Cassandra, TTL 90d)
-- partition by user_id, clustered by created_at desc
-- { notification_id, user_id, channel, template_id, status,
--   attempts, created_at, sent_at, delivered_at, error_code }

The notification log is append-heavy and queried by "this user's recent notifications" — a perfect fit for a wide-column NoSQL store partitioned by user_id with a TTL, not a single relational table that would need constant sharding and purging. Device tokens and preferences are small and read on the hot fan-out path, so they live in a fast store fronted by Redis with a short TTL; a token write (login on a new device) invalidates the cache entry.

Step 12 — Scaling and Fault Tolerance

Every stage scales horizontally and degrades independently.

Scaling

• Workers are stateless — fan-out and channel workers are Kafka consumer groups; add consumers to add throughput, with partitions sized for peak parallelism. Partition by user_id to preserve per-user ordering where it matters (e.g., "shipped" before "delivered").
• Channel pools scale independently — push needs hundreds of workers; SMS needs few because Twilio rate-limits anyway. Autoscale each pool on its own queue depth.
• Provider connection pooling — APNs HTTP/2 connections and FCM batches are reused across sends; a connection pool per worker amortizes TLS/handshake cost.

Fault tolerance

• Provider outage — retries + DLQ absorb it; the affected channel backs up in its queue while others flow. When the provider recovers, drain the backlog and replay the DLQ. Critical: the queue must have enough retention to hold a multi-hour outage's worth of jobs.
• Worker crash — Kafka redelivers uncommitted messages to another consumer; idempotent dedup makes the redelivery harmless.
• Redis (dedup/cache) failure — degrade gracefully: on a dedup-store outage you can fail open (risk a few duplicates) for non-critical channels rather than halt delivery; preferences fall back to the source DB at higher latency.
• Poison messages — a malformed job that always throws is caught by the attempt cap and routed to the DLQ instead of blocking the partition forever.

Step 13 — Observability

You cannot operate a notification system you cannot see into. "Did the user get the OTP?" must be answerable in seconds. Observability is built around the per-notification trace and the delivery funnel.

• End-to-end trace — every notification carries an ID stamped at the API and propagated through the event, fan-out, queue, worker, and provider call, so you can reconstruct its full timeline: accepted → enqueued → sent → provider-acked → delivery-receipt.
• Funnel metrics per channel — accepted, suppressed (prefs/dedup), sent, delivered, failed, opened. A sudden drop in delivered/sent ratio for push is an APNs problem; a spike in suppressed is a preference or dedup anomaly.
• Delivery receipts / webhooks — providers report final status asynchronously (SES bounces, APNs feedback, Twilio status callbacks). Ingesting these closes the loop from "sent" to "actually delivered/bounced" and feeds token pruning and bounce suppression.
• Alerting — alert on DLQ growth rate, per-provider error rate, and queue lag, not just absolute volume. A growing DLQ is the earliest signal of a provider or token problem.

Step 14 — Key Tradeoffs