Distributed Notification Platform: Multi-Channel Delivery, Rate Limiting, Deduplication, and Fan-Out Queues

1. Problem and Motivation

In modern digital products, communication with users is essential: authentication depends on One-Time Passwords (OTPs), marketplaces depend on order confirmations, financial apps depend on fraud alerts, and social platforms depend on real-time activity notifications.

While sending an individual email or push notification via an API SDK (such as SendGrid or Twilio) requires only ten lines of code, orchestrating a centralized notification platform at enterprise scale—dispatching 100 million messages daily across multiple global channels—introduces complex distributed systems challenges:

+---------------------------------------------------------------------------------------------------+
|                              ENTERPRISE NOTIFICATION COMPLEXITIES                                |
+---------------------------------------------------------------------------------------------------+
|  1. Channel Diversity: APNs (Apple), FCM (Google), SMS (Twilio, Infobip), Email (SES, SendGrid),   |
|     In-App WebSockets, and Slack/Teams webhooks each have wildly different protocols and SLAs.    |
|  2. Unpredictable Traffic Surges: A flash sale or breaking news alert enqueues 15,000,000 pushes    |
|     instantaneously, threatening to overwhelm internal brokers and third-party telecom gateways.  |
|  3. Priority Inversion (Head-of-Line Blocking): A marketing broadcast must never block a          |
|     sub-second password reset OTP or emergency security alert.                                    |
|  4. Strict Idempotency: Network retries or message queue re-deliveries must never charge a user's  |
|     credit card twice or send five identical SMS text messages.                                  |
|  5. Downstream Vendor Throttling: External vendors enforce hard rate limits (e.g., Twilio 100/sec,  |
|     SES 500/sec). Exceeding these limits yields HTTP 429 errors, message loss, and penalty fees.  |
|  6. User Preferences & Compliance: GDPR opt-outs, quiet hours (10:00 PM - 08:00 AM in user local   |
|     timezone), and frequency caps (e.g., max 3 marketing emails/week) must be enforced instantly. |
+---------------------------------------------------------------------------------------------------+

2. Mental Model & Architectural Overview

The notification platform decouples Ingestion & Validation, Priority-Based Buffering, Worker Dispatch & Throttling, and Delivery Tracking into independent asynchronous stages.

SYSTEM ARCHITECTURE BLUEPRINTAPPLICATION ARCHITECTURES

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Upstream Services & Deduplication Gate

Establishes the ingress and structural boundary across Upstream Services & Deduplication Gate: Auth, Orders, Marketing POST /api/v1/notify; Redis SETNX (dedup_key, TTL) Suppresses duplicate retries; Window: 10m - 24h.

Architectural Invariant: Record initial status: PENDING.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. P1 Queue: Transactional & P2 Queue: Bulk Marketing

Coordinates core data-plane execution and state transitions through P1 Queue: Transactional & P2 Queue: Bulk Marketing: SLA: < 30 seconds; Order confirmation, payments; Auto-scaling worker group.

Architectural Invariant: Pre-warmed connection pools Never blocked by marketing.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Dead Letter Queue (DLQ) & Apple APNs & Google FCM

Enforces durability, state convergence, and output verification across Dead Letter Queue (DLQ) & Apple APNs & Google FCM: Max 5 retries exceeded Invalid device tokens / dead emails Auto-unregisters dead APNs tokens; HTTP/2 Persistent Multiplexing Push Notifications; SMPP / HTTP REST API Dynamic Carrier Failover.

Architectural Invariant: ClickHouse / OpenTelemetry Tracks delivery latencies & carrier error rates.

The Core Stages:

1. Ingestion API & Filter Layer:
   • Upstream services (Auth, Checkout, Social Feed) submit notification requests via POST /api/v1/notify.
   • Distributed Deduplication Gate: Computes an idempotency hash and checks Redis using an atomic SETNX lease. If a duplicate is detected within the deduplication window (e.g., 10 minutes), the request is dropped immediately.
   • User Preference & Quiet Hours Engine: Queries the user preference cache. If the user opted out of the channel, or if the current timestamp falls within the user’s localized quiet hours (and the message is not high-priority), the message is either suppressed or rescheduled into a delayed queue.
   • Priority Classification: Tags the message with a strict priority tier ($P_0$, $P_1$, $P_2$).
2. Partitioned Priority Queues:
   • Instead of a single shared FIFO queue, the platform partitions traffic into separate queues:
     • $P_0$ (Urgent / Security): Auth OTPs, fraud alerts, password resets. Target SLA: $<2\text{ seconds}$.
     • $P_1$ (Transactional): Order confirmations, flight changes, direct chat messages. Target SLA: $<30\text{ seconds}$.
     • $P_2$ (Promotional / Bulk): Marketing campaigns, weekly digests, re-engagement emails. Target SLA: $<4\text{ hours}$.
3. Dispatch Workers & Distributed Rate Limiting:
   • Independent auto-scaling worker pools consume from each queue tier.
   • Before invoking downstream third-party APIs, workers query a Distributed Token Bucket Rate Limiter (backed by Redis Lua scripts) to guarantee cluster-wide compliance with vendor transactions-per-second (TPS) contracts.
4. Carrier Adapters, Circuit Breakers & Dynamic Failover:
   • Normalized vendor adapters translate payloads into vendor-specific HTTP/2 or SMPP formats.
   • A Circuit Breaker monitors vendor error rates ($5\text{xx}$ spikes or carrier latencies). If Primary SMS Vendor (e.g., Twilio) degrades, traffic fails over automatically to Secondary SMS Vendor (e.g., MessageBird or Infobip).
5. Reconciliation & Status Webhooks:
   • Downstream providers send asynchronous HTTP webhooks reporting delivery status (DELIVERED, BOUNCED, REJECTED).
   • A webhook ingestion gateway reconciles the delivery lifecycle in distributed storage (Cassandra / DynamoDB) and automatically deactivates invalid push tokens.

Architectural Blueprint: Distributed Task Scheduler (Hierarchical Timing Wheels vs. SKIP LOCKED)

Delayed and recurring notifications (e.g., “send reminder in 25 minutes”, “execute billing cron every hour”) require a fault-tolerant Distributed Task Scheduler:

1. Database-Backed Work Claiming (SELECT ... FOR UPDATE SKIP LOCKED):
   • For schedules up to ~$10,000\text{ jobs/sec}$, sharded PostgreSQL tables indexed on (status, scheduled_at) allow stateless worker nodes to claim due tasks without lock contention:

     UPDATE scheduled_jobs
     SET status = 'RUNNING', locked_by = $1, locked_until = NOW() + INTERVAL '60 seconds'
     WHERE job_id IN (
       SELECT job_id FROM scheduled_jobs
       WHERE status = 'PENDING' AND scheduled_at <= NOW()
       ORDER BY scheduled_at ASC LIMIT 100
       FOR UPDATE SKIP LOCKED
     ) RETURNING *;

   • SKIP LOCKED immediately skips rows currently locked by peer workers, turning a relational table into a high-concurrency lock-free work queue.
2. In-Memory Hierarchical Timing Wheels ($O(1)$ Insertion & Cancellation):
   • Tracking millions of active timers in a min-heap costs $O(\log N)$ per insert/cancel. Hashed & Hierarchical Timing Wheels (Varghese & Lauck, used in Kafka, Netty, and Linux kernel timers) organize time into circular ring slots (e.g., Wheel 1: $60\times 1\text{ s}$ slots; Wheel 2: $60\times 1\text{ min}$ slots; Wheel 3: $24\times 1\text{ hr}$ slots) backed by an append-only WAL.
   • Advancing the current tick pointer costs $O(1)$ amortized time, cascading overflowing high-horizon buckets down into finer wheels as deadlines approach.

3. Terminology & Core Concepts

• APNs (Apple Push Notification service): Apple’s binary HTTP/2 multiplexed protocol for sending push notifications to iOS and macOS devices.
• FCM (Firebase Cloud Messaging): Google’s cross-platform messaging service for Android, iOS, and Web.
• SMPP (Short Message Peer-to-Peer): An industry-standard telecommunications protocol used by SMS aggregators to exchange SMS messages directly with mobile network operators.
• Idempotency Key: A unique deterministic string derived from the event metadata used to guarantee that identical requests received multiple times produce exactly one external action.
• Head-of-Line (HoL) Blocking: A performance hazard where a queue is monopolized by a massive slow task (e.g., a 10M newsletter blast), preventing subsequent urgent tasks (e.g., login OTPs) from being processed.
• Token Bucket Algorithm: A rate-limiting algorithm that maintains a bucket of tokens refilled at a constant rate $r$; each request consumes 1 token. It allows bursts up to the bucket capacity $b$ while enforcing an average rate limit $r$.
• Dead Letter Queue (DLQ): A dedicated queue holding messages that have repeatedly failed processing after exhaustive retries, isolating poison-pill messages for engineering inspection.

4. Architectural Deep Dive: Deduplication & Priority Isolation

1. Distributed Deduplication (Preventing Double-Dispatch)

Distributed message brokers (Kafka, RabbitMQ, Amazon SQS) operate under At-Least-Once delivery semantics. If a network blip occurs after a worker finishes sending an SMS but before it commits the queue offset, the message broker re-delivers the message to a different worker.

Without deduplication, the customer receives two identical SMS messages and the company pays double telecom fees.

Incoming Request
      |
      v
+-----------------------------------------------------------+
| Deduplication Key: SHA256(user_id + channel + event_hash) |
+-----------------------------------------------------------+
      |
      v
+-----------------------------------------------------------+
| Redis: SET idempotency:key "LOCKED" NX EX 600             |
+-----------------------------------------------------------+
      |                                   |
      | Key Acquired (1)                  | Key Exists (0)
      v                                   v
+-----------------------+       +------------------------------------+
| Dispatch Notification |       | Duplicate Detected: Drop / ACK OK  |
+-----------------------+       +------------------------------------+

The deduplication lease uses an atomic Redis command:

SET notify:dedup:<user_id>:<channel>:<template_id>:<hash> "IN_FLIGHT" NX EX 600

• If Redis returns OK: The worker successfully acquired the unique execution lease.
• If Redis returns nil: Another worker is currently processing or has already completed this dispatch within the 600-second window. The worker drops the duplicate and acknowledges the broker.

2. Design Alternatives & Trade-Off Matrix: Queue Isolation, Deduplication, and Failover

Consider what happens when a single queue is used during a Black Friday promotional campaign:

$$\text{Marketing Blast}=10,000,000\text{ push notifications enqueued at }09:00\text{ AM}$$ $$\text{Worker Dispatch Capacity}=5,000\text{ dispatches/sec}$$ $$\text{Queue Drainage Time}=\frac{10,000,000}{5,000}=2,000\text{ seconds}\approx 33.3\text{ minutes}$$

If User Bob attempts to log in at 09:05 AM ($300\text{ seconds}$ into the blast, after $300\times 5,000=1,500,000$ messages have drained) and requests a Two-Factor Authentication OTP, his OTP message is placed behind the remaining $8,500,000$ items in the queue. Bob’s OTP will wait $\frac{8,500,000}{5,000}=1,700\text{ seconds}\approx 28.3\text{ minutes}$—guaranteeing authentication failure.

+---------------------------------------------------------------------------------------------------+
|                                  PRIORITY TIER ISOLATION TOPOLOGY                                 |
+---------------------------------------------------------------------------------------------------+
|  [P0: Urgent Queue]  -----> Dedicated Pool 0 (50 pods)   -----> Target SLA: < 2 sec (OTP / Auth)  |
|  [P1: Normal Queue]  -----> Dedicated Pool 1 (100 pods)  -----> Target SLA: < 30 sec (Orders)     |
|  [P2: Bulk Queue]    -----> Auto-scaled Pool 2 (20 pods)  -----> Target SLA: < 4 hrs (Marketing)  |
+---------------------------------------------------------------------------------------------------+

By decoupling into distinct physical queues with dedicated worker pools, $P_0$ queues remain permanently near-zero depth regardless of $P_2$ marketing volume.

5. Mathematical Models: Token Bucket Dynamics, Retry Jitter & Queue Sizing

1. Token Bucket Rate Limiter Formulation

Telecommunication gateways (e.g., Twilio, Amazon SES) strictly penalize rate spikes. Let:

• $r$: Token generation rate (tokens per second)
• $b$: Maximum token bucket burst capacity
• $C(t)$: Number of available tokens at time $t$
• $\Delta t=t-t_{\text{last}}$: Elapsed time since last token replenishment

The available token capacity is updated via: $$C(t)=\min (b,C(t_{\text{last}})+\Delta t\cdot r)$$

If $C(t)\ge 1$: The token request succeeds, $C(t)\leftarrow C(t)-1$, and dispatch proceeds. Otherwise, the request is delayed or rejected with backoff.

Worked Example: SMS Gateway Throttling

A telecommunications aggregator permits an average rate of $r=200\text{ SMS/sec}$ with an allowable burst $b=500\text{ SMS}$. Assuming the bucket starts full ($C(0)=b=500$), if an upstream microservice abruptly dumps $N=2,000$ SMS messages simultaneously into the worker pool:

1. The first $b=500$ messages consume the pre-accumulated burst tokens and are dispatched immediately ($t\approx 0$).
2. The remaining $N-b=1,500$ messages must wait for tokens to replenish at the steady rate $r=200\text{ SMS/sec}$.
3. Total time to clear the burst: $$T=\frac{N-b}{r}=\frac{2,000-500}{200}=\frac{1,500}{200}=7.5\text{ seconds}$$

2. Queue Backlog & Sizing Derivation

Let $\lambda (t)$ be the arrival rate of notifications and $\mu$ be the service rate per worker node. For a cluster of $M$ active worker pods: $$\text{Total Processing Capacity }S=M\cdot \mu$$

The instantaneous queue growth rate is: $$\frac{dQ}{dt}=\lambda (t)-S$$

If a flash marketing campaign generates an arrival spike of $\lambda =50,000\text{ messages/sec}$ for $T_{\text{spike}}=60\text{ seconds}$, and our downstream vendor rate limiter caps processing at $S=10,000\text{ messages/sec}$: $$\text{Peak Queue Backlog }{Q}_{\text{max}}=(\lambda -S)\cdot T_{\text{spike}}=(50,000-10,000)\times 60=2,400,000\text{ messages}$$

If each serialized message payload consumes $1.2\text{ KB}$ ($1,200\text{ bytes}$, decimal) in Apache Kafka: $$\text{Buffer Storage Required}=2.4\times 10^6\times 1,200\text{ bytes}=2.88\times 10^9\text{ bytes}=2.88\text{ GB}(\approx 2.68\text{ GiB})$$

Once the 60-second campaign burst finishes and background traffic returns to ${\lambda }_{\text{normal}}=2,000\text{ messages/sec}$, the backlog drains at net rate $S-{\lambda }_{\text{normal}}=8,000\text{ messages/sec}$, clearing in $T_{\text{drain}}=\frac{2,400,000}{8,000}=300\text{ seconds}$ ($5\text{ minutes}$). This proves why Apache Kafka’s sequential disk-backed log architecture is essential: pure in-memory queues risk out-of-memory (OOM) eviction under multi-million message backlogs.

3. Exponential Backoff with Full Jitter

When downstream carriers return transient HTTP 429 or HTTP 503 responses, deterministic exponential backoff ($T_{\text{base}}\cdot 2^k$) causes all throttled workers to sleep for the exact same duration and collide again in synchronized retry waves. Production workers draw a uniform random wait using Full Jitter:

where $k\in \{0,1,2,\dots \}$ is the retry attempt index, $T_{\text{base}}$ is initial backoff (e.g., $200\text{ ms}$), and $T_{\text{cap}}$ is maximum delay (e.g., $30\text{ s}$). At attempt $k=4$, the window ceiling is $\min (30\text{ s},0.2\text{ s}\times 16)=3.2\text{ s}$ with expected wait $\mathbb{E}[T_{\text{sleep}}(4)]=1.6\text{ s}$. Across $10,000$ throttled notifications, Full Jitter spreads retries uniformly over $[0,3.2\text{ s}]$ ($\approx 3,125\text{ retries/sec}$) rather than firing an instantaneous $10,000$-request spike at $t=3.2\text{ s}$.

6. Dynamic Carrier Failover & Circuit Breakers

SMS delivery across global telecommunications is notoriously brittle: regional telecom carriers frequently experience fiber cuts, routing errors, or regulatory filtering.

An enterprise notification engine implements an automated Three-State Circuit Breaker per downstream vendor:

                  +-------------------------+
                  |                         |
                  |         CLOSED          |  (Normal Operation: 100% Traffic to Vendor A)
                  |                         |
                  +-------------------------+
                               |
                               | Error rate > 15% or Latency > 2500ms
                               v
                  +-------------------------+
                  |                         |
                  |          OPEN           |  (Tripped: 100% Traffic routed to Vendor B)
                  |                         |
                  +-------------------------+
                               |
                               | Cooldown period elapsed (60 seconds)
                               v
                  +-------------------------+
                  |                         |
                  |        HALF-OPEN        |  (Probe: Send 5% trial traffic to Vendor A)
                  |                         |
                  +-------------------------+
                     /                   \
    Success rate >= 98%                 Probe fails
                   /                       \
                  v                         v
               CLOSED                      OPEN

7. Production Implementation: Resilient Notification Engine

The following complete TypeScript implementation demonstrates:

1. Distributed deduplication with Redis SETNX.
2. Priority queue router.
3. Redis Lua-based Token Bucket rate limiting.
4. Circuit Breaker carrier failover.

/**
 * Production Distributed Notification Engine
 * Deduplication, Token Bucket Limiting, and Carrier Failover.
 */

export type Priority = 'P0_URGENT' | 'P1_TRANSACTIONAL' | 'P2_BULK';
export type Channel = 'PUSH' | 'SMS' | 'EMAIL';

export interface NotificationPayload {
  notificationId: string;
  userId: string;
  channel: Channel;
  priority: Priority;
  recipient: string; // phone number, email, or device token
  templateId: string;
  templateArgs: Record<string, string>;
  idempotencyWindowSeconds?: number;
}

export interface DeliveryResult {
  status: 'DELIVERED' | 'DUPLICATE_SUPPRESSED' | 'THROTTLED' | 'FAILED';
  provider: string;
  latencyMs: number;
  messageId?: string;
  error?: string;
}

/**
 * Distributed Token Bucket Rate Limiter
 */
export class TokenBucketLimiter {
  private tokens: number;
  private lastRefillTimestamp: number;

  constructor(
    private readonly capacity: number,
    private readonly refillRatePerSecond: number
  ) {
    this.tokens = capacity;
    this.lastRefillTimestamp = Date.now();
  }

  public tryAcquire(requestedTokens = 1): boolean {
    const now = Date.now();
    const elapsedSeconds = (now - this.lastRefillTimestamp) / 1000;
    this.lastRefillTimestamp = now;

    // Replenish tokens up to max capacity
    this.tokens = Math.min(
      this.capacity,
      this.tokens + elapsedSeconds * this.refillRatePerSecond
    );

    if (this.tokens >= requestedTokens) {
      this.tokens -= requestedTokens;
      return true;
    }
    return false; // Throttled
  }
}

/**
 * Three-State Circuit Breaker for Carrier Gateways
 */
export class CarrierCircuitBreaker {
  private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';
  private failureCount = 0;
  private successCount = 0;
  private lastStateChange = Date.now();

  constructor(
    public readonly name: string,
    private readonly failureThreshold = 5,
    private readonly recoveryTimeMs = 30_000
  ) {}

  public canExecute(): boolean {
    const now = Date.now();
    if (this.state === 'OPEN') {
      if (now - this.lastStateChange >= this.recoveryTimeMs) {
        this.state = 'HALF_OPEN';
        this.lastStateChange = now;
        return true; // Allow trial probe
      }
      return false; // Fast fail
    }
    return true;
  }

  public recordSuccess(): void {
    if (this.state === 'HALF_OPEN') {
      this.successCount += 1;
      if (this.successCount >= 3) {
        this.state = 'CLOSED';
        this.failureCount = 0;
        this.successCount = 0;
      }
    } else {
      this.failureCount = 0;
    }
  }

  public recordFailure(): void {
    this.failureCount += 1;
    this.lastStateChange = Date.now();
    if (this.failureCount >= this.failureThreshold || this.state === 'HALF_OPEN') {
      this.state = 'OPEN';
    }
  }

  public getState(): string {
    return this.state;
  }
}

/**
 * Enterprise Notification Dispatch Service
 */
export class NotificationDispatcher {
  constructor(
    private readonly redisMock: Map<string, string>,
    private readonly rateLimiter: TokenBucketLimiter,
    private readonly primaryCarrierBreaker: CarrierCircuitBreaker,
    private readonly fallbackCarrierBreaker: CarrierCircuitBreaker
  ) {}

  public async dispatch(payload: NotificationPayload): Promise<DeliveryResult> {
    const start = Date.now();

    // 1. Distributed Deduplication Check (SETNX simulation)
    const dedupKey = `dedup:${payload.userId}:${payload.channel}:${payload.templateId}`;
    if (this.redisMock.has(dedupKey)) {
      return {
        status: 'DUPLICATE_SUPPRESSED',
        provider: 'NONE',
        latencyMs: Date.now() - start,
      };
    }

    // 2. Token Bucket Rate Limiter Check (before persisting dedupKey so throttled retries are not dropped)
    if (!this.rateLimiter.tryAcquire(1)) {
      return {
        status: 'THROTTLED',
        provider: 'NONE',
        latencyMs: Date.now() - start,
        error: 'Downstream vendor TPS ceiling reached. Re-queueing with backoff.',
      };
    }

    // 3. Carrier Selection & Circuit Breaker Execution
    let selectedCarrier = 'PrimaryCarrier';
    let breaker = this.primaryCarrierBreaker;

    if (!breaker.canExecute()) {
      // Primary carrier is OPEN (degraded) -> Failover to secondary
      selectedCarrier = 'SecondaryCarrier';
      breaker = this.fallbackCarrierBreaker;
      if (!breaker.canExecute()) {
        return {
          status: 'FAILED',
          provider: 'ALL_CIRCUITS_OPEN',
          latencyMs: Date.now() - start,
          error: 'Both primary and secondary telecom gateways are down.',
        };
      }
    }

    // Lock deduplication key in Redis once rate limiter and circuit breaker admit the request
    this.redisMock.set(dedupKey, payload.notificationId);

    // 4. Simulate External Carrier Network Call
    try {
      const messageId = await this.sendToVendor(selectedCarrier, payload);
      breaker.recordSuccess();
      return {
        status: 'DELIVERED',
        provider: selectedCarrier,
        messageId,
        latencyMs: Date.now() - start,
      };
    } catch (err) {
      this.redisMock.delete(dedupKey);
      breaker.recordFailure();
      return {
        status: 'FAILED',
        provider: selectedCarrier,
        error: (err as Error).message,
        latencyMs: Date.now() - start,
      };
    }
  }

  private async sendToVendor(
    carrier: string,
    payload: NotificationPayload
  ): Promise<string> {
    // In production: HTTP/2 call to APNs, or REST call to Twilio/SendGrid
    return `msg_${carrier}_${Date.now()}_${payload.notificationId.slice(0, 8)}`;
  }
}

async function runDemo(): Promise<void> {
  console.log("=== Notification Platform: Priority Queue & Circuit Breakers ===");
  const redisMock = new Map<string, string>();
  const rateLimiter = new TokenBucketLimiter(10, 5); // 10 capacity, 5 refill/sec
  const primaryBreaker = new CarrierCircuitBreaker("sendgrid", 2, 5000);
  const fallbackBreaker = new CarrierCircuitBreaker("amazon-ses", 2, 5000);
  const dispatcher = new NotificationDispatcher(redisMock, rateLimiter, primaryBreaker, fallbackBreaker);
  console.log("Dispatcher initialized with Token Bucket limiter and SendGrid/SES circuit breakers");

  const payload: NotificationPayload = {
    notificationId: "notif-101",
    userId: "user-42",
    channel: "EMAIL",
    priority: "P1_TRANSACTIONAL",
    recipient: "alice@example.com",
    templateId: "welcome-email",
    templateArgs: { firstName: "Alice" },
    idempotencyWindowSeconds: 600,
  };

  const first = await dispatcher.dispatch(payload);
  console.log(`First dispatch: status=${first.status}, provider=${first.provider}, breaker=${primaryBreaker.getState()}`);

  const duplicate = await dispatcher.dispatch(payload);
  console.log(`Retry dispatch: status=${duplicate.status}, provider=${duplicate.provider}`);

  primaryBreaker.recordFailure();
  primaryBreaker.recordFailure();
  const failoverPayload: NotificationPayload = {
    ...payload,
    notificationId: "notif-102",
    templateId: "otp-login",
    priority: "P0_URGENT",
  };
  const failoverResult = await dispatcher.dispatch(failoverPayload);
  console.log(`Failover dispatch: status=${failoverResult.status}, provider=${failoverResult.provider}, primaryBreaker=${primaryBreaker.getState()}`);
}

runDemo();

8. Asynchronous Delivery Reconciliation & Webhooks

Transmitting a notification to a downstream provider only confirms that the provider accepted the payload (HTTP 202 Accepted). It does not confirm that the SMS reached the cell tower or that the push notification rendered on the iPhone screen.

+---------------------------------------------------------------------------------------------------+
|                                 DELIVERY LIFECYCLE STATE MACHINE                                  |
+---------------------------------------------------------------------------------------------------+
|  [SUBMITTED]  ----->  [ENQUEUED]  ----->  [SENT_TO_GATEWAY]  ----->  [DELIVERED_TO_DEVICE]        |
|                                                                              |                    |
|                                                                              v                    |
|                                                                        [CLICKED / OPENED]         |
|  Failures:                                                                                        |
|  [SENT_TO_GATEWAY]  ------------------->  [BOUNCED / FAILED]  ----->  [DLQ / TOKEN DEACTIVATED]   |
+---------------------------------------------------------------------------------------------------+

Ingestion of Carrier Webhooks:

1. Downstream carriers dispatch HTTP POST callbacks to our https://webhooks.example.com/carrier/status.
2. A high-throughput gateway validates the cryptographic signature (HMAC-SHA256) of the webhook.
3. The event is written directly to Kafka topic notification-delivery-receipts.
4. Consumer workers update the message state in Cassandra:

   UPDATE notification_events
   SET status = 'DELIVERED',
       carrier_latency_ms = 412,
       delivered_at = toTimestamp(now())
   WHERE message_id = 'msg_Twilio_1761899120';

5. If the webhook reports BadDeviceToken (APNs HTTP 410) or HardBounce (SES 550), the system dispatches an event to the User Device Service to deactivate the invalid token immediately.

9. Failure Modes, Edge Cases & Threat Modeling

10. Knowledge Check

Review the core architectural principles, then complete the 4-question interactive assessment quiz at the end of this lesson:

1. Queue Isolation: Articulate why Head-of-Line blocking in shared queues destroys transactional OTP authentication SLAs.
2. Idempotency Leases: Explain how Redis atomic SETNX prevents duplicate SMS transmissions during Kafka consumer rebalances.
3. Token Bucket Math: Derive how burst capacity $b$ and replenishment rate $r$ shape traffic to match downstream vendor contracts.
4. Circuit Breaker States: Trace the state transitions from CLOSED to OPEN to HALF_OPEN during carrier degradation.

11. Practical Exercises & Deliverables

Exercise 1: Implement Redis Lua Token Bucket

Problem: Traditional client-side rate limiting fails in distributed clusters because 50 independent worker pods cannot synchronize local token counters without race conditions.
Deliverable:

1. Write an atomic Redis Lua script implementing the Token Bucket algorithm (KEYS[1] = bucket key, ARGV[1] = capacity, ARGV[2] = refill rate per sec, ARGV[3] = current timestamp).
2. Benchmark the script using ioredis under 5,000 concurrent requests. Prove that the total allowed requests never exceed the specified capacity and replenishment rate.

Exercise 2: Simulate Carrier Degradation & Circuit Breaker Failover

Problem: Test how your notification pipeline responds when the primary SMS vendor fails.
Deliverable:

1. Create a mock vendor client that injects simulated network timeouts and HTTP 503 errors on 25% of requests.
2. Wrap the vendor client in the three-state Circuit Breaker implemented in Section 7.
3. Verify that after 5 consecutive failures, the breaker trips to OPEN, routing 100% of subsequent dispatches to the secondary vendor with $<1\text{ ms}$ failover latency.

12. Summary & Architectural Decision Guide

1. Physically Partition Priority Tiers: Never multiplex $P_0$ authentication OTPs and $P_2$ promotional campaigns on the same broker queue or worker pool; physical topic and consumer isolation prevents multi-million-message Head-of-Line blocking.
2. Pair At-Least-Once Queues with Atomic Idempotency Leases: Acquire a deterministic Redis SET ... NX EX deduplication lease before invoking external vendor APIs so consumer crashes and Kafka rebalances never double-send SMS or email messages.
3. Shape Cluster Egress with Token Buckets and Full Jitter: Enforce contractual provider TPS limits via centralized Redis Lua Token Buckets, and handle transient HTTP 429/503 responses with Full-Jitter exponential backoff.
4. Close the Loop via Signed Webhooks: Treat HTTP 202 Accepted as handoff, not delivery; ingest HMAC-verified carrier callbacks asynchronously to reconcile terminal states and prune HTTP 410 BadDeviceToken registrations.

13. Authoritative References & Version Context

• Apple Developer Documentation — Apple Push Notification service (APNs) HTTP/2 Provider API: Sending Notification Requests to APNs (HTTP/2 multiplexed provider protocol and 410 Unregistered token lifecycle semantics).
• Google Firebase Cloud Messaging (FCM) — HTTP v1 API Specification: FCM Architectural Overview & HTTP v1 Protocol (FCM v1 OAuth2 dispatch, topic/token fan-out, and quota error handling).
• Marc Brooker (AWS Builders’ Library) — Exponential Backoff and Jitter: Timeouts, Retries, and Backoff with Jitter (mathematical analysis of Full Jitter and Decorrelated Jitter under downstream contention).
• IETF RFC 6585 — Additional HTTP Status Codes (429 Too Many Requests): RFC 6585 Section 4 (Retry-After semantics and rate-limiting response contracts; baseline stack context: Apache Kafka 3.7+, Redis 7.2+, Apache Cassandra 4.1+/5.0).