Real-Time Chat & Presence Systems: WebSockets, Message Ordering, and Ephemeral Heartbeats

1. The Core Engineering Challenge: Stateful Scale

Stateless web architectures scale with relative ease: when traffic surges, an orchestrator launches additional container replicas behind an HTTP load balancer. Any container can handle any incoming request because state resides in an external database or cache cluster.

Real-time communication platforms (Slack, Discord, WhatsApp, Microsoft Teams, WeChat) completely break this assumption:

1. Persistent Stateful Connections: Clients do not initiate transient request-response round-trips. Instead, hundreds of millions of client devices maintain long-lived TCP/TLS sockets (WebSockets) that remain open for hours or days.
2. Reverse Delivery (Server Push): When User A sends a message to User B, the server must push that payload down User B’s open socket within sub-100 ms, regardless of which physical server across a global cluster terminates User B’s connection.
3. Strict Causal & Conversational Ordering: If Alice asks “Can you deploy to production?” and Bob replies “No, don’t do it!”, delivering those messages out of order completely inverts conversational meaning.
4. Extreme Write Concurrency with High Fan-Out: Unlike social networks where reads vastly exceed writes ($99:1$), active chat platforms frequently exhibit $1:1$ write-to-read ratios in direct messaging, and massive $1:100,000$ fan-out bursts in public broadcast channels.
5. Transient Connectivity & Flaky Radios: Mobile devices constantly roam between cell towers, enter subway tunnels, drop packets, sleep to conserve battery, and reconnect with stale state.

+-------------------------------------------------------------------------------+
|                       THE DUAL PARADIGM COMPARISON                             |
+-------------------------------------------------------------------------------+
| Dimension            | Stateless Web Tier (HTTP)   | Real-Time Chat (WebSocket)|
+----------------------+-----------------------------+---------------------------+
| Connection Lifetime  | Ephemeral (milliseconds)    | Persistent (hours / days) |
| Protocol Overhead    | High (800B headers/request) | Minimal (2-6B frame header|
| Communication        | Client pull (Request-Reply) | Full-duplex Bi-directional|
| Server Topology      | Any server handles any call | State pinned to Gateway   |
| Memory Profile       | Recycled on request end     | Pinned socket buffers/conn|
| Scaling Model        | Horizontal Auto-Scaling     | Stateful Session Sharding |
+-------------------------------------------------------------------------------+

2. Mental Model: The Distributed Gateway & Session Mesh

To route messages across an elastic fleet of stateful servers, modern instant messaging platforms decouple the Connection Edge from the Business Logic & Storage Plane.

SYSTEM ARCHITECTURE BLUEPRINTAPPLICATION ARCHITECTURES

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. User A (Sender - Mobile) & WebSocket Gateway Cluster

Establishes the ingress and structural boundary across User A (Sender - Mobile) & WebSocket Gateway Cluster: Persistent TLS Socket (WSS) SEND { chan: "c_1", msg: "Hi" }; Epoll/Kqueue event loops Terminates 1M+ idle TCP sockets; Gateway Node 1: User A socket.

Architectural Invariant: Active connection to Gateway Node 2 RECV { seq: 1042, msg: "Hi" }.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. User C (Receiver - Offline) & Distributed Presence Engine

Coordinates core data-plane execution and state transitions through User C (Receiver - Offline) & Distributed Presence Engine: Disconnected / Mobile sleep Requires APNS/FCM Push & Delta Sync; Heartbeat ping every 25s (TTL 30s) Redis SET user:A:gw gw_1 EX 30 ZADD presence_set <now> user:A; Per-Channel Monotonic Ordering Snowflake ID + Raft/Redis counter seq_id = INCR channel:c_1:seq.

Architectural Invariant: PUBLISH gw:gw_2 {payload} Offline routing trigger: XADD offline_stream {user_C, seq}.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Hot Timeline Cache (Redis) & Cold Append-Only Store

Enforces durability, state convergence, and output verification across Hot Timeline Cache (Redis) & Cold Append-Only Store: Recent 1,000 messages per room ZADD room:c_1 1042 "{msg_bytes}" Sub-millisecond instant fetch; ScyllaDB / Cassandra LSM Cluster; Partition Key: (channel_id, bucket).

Architectural Invariant: User C reconnects with Cursor: last_delivered_seq = 1010 Fetch delta [1011..1042] in 1 call Guarantees zero dropped messages.

The architecture divides responsibilities across four distinct layers:

1. WebSocket Gateway Tier (Stateful Edge):
   • Maintains open TCP/TLS connections with client devices using non-blocking I/O multiplexing (epoll on Linux, kqueue on BSD/macOS).
   • Responsible strictly for TLS termination, frame serialization/deserialization, heartbeat detection, and socket read/write pumping.
   • Contains zero persistent business logic. If a gateway node crashes, clients immediately reconnect to another gateway node.
2. Session & Presence Registry (Distributed In-Memory Directory):
   • Tracks the dynamic mapping of every connected user_id to their active gateway_node_id and connection_id.
   • Backed by an in-memory datastore (such as Redis Cluster) using fast hash maps and sorted sets.
3. Message Sequencer & Broker Plane (Ordering & Routing):
   • Assigns strictly monotonic, tamper-proof sequence IDs to messages on a per-channel basis.
   • Routes outgoing messages to the specific gateway nodes that hold active sockets for channel recipients via a distributed pub/sub bus (e.g., Redis Streams, Apache Kafka, or NATS).
4. Dual-Tier Storage Plane:
   • Hot Timeline Cache: Redis Sorted Sets holding the recent $1,000$ messages per active room for sub-millisecond retrieval.
   • Cold Append-Only Historical Store: Wide-column NoSQL databases (ScyllaDB / Apache Cassandra) optimized for high-throughput sequential disk writes and chronological range queries.

Design Alternatives & Protocol Trade-Off Matrix

Architectural Blueprint: Distributed Web Crawler (URL Frontier, Politeness & SimHash Dedup)

When chat and social platforms generate link previews or index public web pages at scale, they rely on a Distributed Web Crawler:

1. Two-Tier URL Frontier (Priority + Politeness Routing):
   • Front Queues (Priority): Rank discovered URLs by PageRank, domain authority, and update frequency into $P$ priority queues.
   • Back Queues (Politeness): Each back queue corresponds to a single target hostname (example.com) and is pinned to one fetcher worker via consistent hashing. A min-heap tracks the next permitted fetch timestamp $t_{\text{next}}=t_{\text{last\_response}}+\max (\text{Crawl-Delay},10\times {\tau }_{\text{fetch}})$, guaranteeing strict robots.txt politeness and zero concurrent requests to the same origin server.
2. Spider Traps & Near-Duplicate Content Detection (SimHash):
   • URL Canonicalization & Bloom Filters: Normalize query parameters, strip session IDs, enforce maximum path depth, and check a partitioned Bloom filter (or RocksDB keyset) before enqueueing.
   • 64-Bit Charikar SimHash for Near-Duplicates: Cryptographic hashes (SHA-256) change completely on a 1-byte timestamp edit. SimHash hashes document 3-gram tokens into a 64-dimensional weighted vector and collapses signs into a 64-bit fingerprint. Two pages are near-duplicates if their Hamming distance $d_H(\text{SimHash}(A),\text{SimHash}(B))\le 3$, detected in $O(1)$ via 4-band Pigeonhole table lookups.

3. Mathematical & Memory Model: Gateway Sizing

Terminating hundreds of thousands of concurrent WebSocket connections per server requires strict operating system tuning and memory accounting.

Socket Memory Breakdown

A single idle WebSocket connection consumes memory across three layers:

1. Kernel TCP Socket Buffers:
   • rmem (Receive buffer): Default Linux TCP receive buffer (typically $4\text{ KB}-8\text{ KB}$).
   • wmem (Transmit buffer): Default Linux TCP send buffer (typically $4\text{ KB}-16\text{ KB}$).
2. TLS / Crypto Overhead: OpenSSL / BoringSSL session state and encryption cipher contexts ($\approx 2\text{ KB}-4\text{ KB}$ per TLS socket).
3. Application Connection State: In-memory data structures tracking user metadata, authentication tokens, subscribe channels, and pending write frame queues ($\approx 2\text{ KB}-6\text{ KB}$).

Gateway Capacity Calculation

If an enterprise cluster must support $10,000,000$ concurrently connected users (using decimal $1\text{ KB}=1,000\text{ bytes}$):

If deploying cloud instances with $64\text{ GB}$ of RAM, dedicating $45\text{ GB}$ to user socket buffers and leaving $19\text{ GB}$ for OS buffers and garbage collection:

Linux Kernel File Descriptor & Epoll Limits

By default, Linux limits a single process to $1,024$ file descriptors (ulimit -n). Terminating $1,000,000$ sockets on a single server requires explicit system configuration:

# /etc/sysctl.conf
# Maximum file descriptors across entire system
fs.file-max = 2097152

# Allocate memory for TCP read/write buffers: min, default, max
net.ipv4.tcp_rmem = 2048 4096 16384
net.ipv4.tcp_wmem = 2048 4096 16384

# Enable rapid reuse of TIME_WAIT sockets
net.ipv4.tcp_tw_reuse = 1

# Maximum TCP listen backlog for incoming handshakes
net.core.somaxconn = 65535

4. Monotonic Message Ordering: Per-Channel Sequencers

In distributed systems, physical timestamps ($T_{\text{NTP}}$) cannot be trusted to order messages (as demonstrated in CORE-M17-002). Two users sending messages from smartphones with even a $50\text{ ms}$ clock skew will experience reversed conversations if sorted by device wall-clock time.

Device Wall-Clock Ordering Failure:
---------------------------------------------------------------------------------
User A (Clock: 12:00:00.040): "Can we delete database table users?"
User B (Clock: 12:00:00.010 [Lagging by 50ms]): "NO! STOP!"

If sorted by Client Timestamp:
1. [12:00:00.010] User B: "NO! STOP!"
2. [12:00:00.040] User A: "Can we delete database table users?"
-> Catastrophic context inversion!

The Per-Channel Total Order Invariant

To guarantee conversational consistency, chat platforms enforce a fundamental invariant:

Because conversational causality only matters within the same channel or conversation, we do not need a globally serialized clock across the entire planet. We need a strictly monotonic sequence counter per channel.

+-------------------------------------------------------------------------------+
|                      MESSAGE SEQUENCING ARCHITECTURES                         |
+-------------------------------------------------------------------------------+
| Strategy                 | Ordering Guarantee | Scalability | Complexity      |
+--------------------------+--------------------+-------------+-----------------+
| 1. Centralized SQL Auto  | Strictly Monotonic | Horrible    | Low (Locks/RPS  |
|    Increment (e.g. MySQL)| (1, 2, 3, 4...)    | (<5k ops/s) | bottlenecks)    |
|                          |                    |             |                 |
| 2. Distributed Snowflake | K-Sorted by time   | Infinite    | Low (Worker ID  |
|    (64-bit ID generator) | (Weak causality)   | (>10M ops/s)| coordination)   |
|                          |                    |             |                 |
| 3. Channel Sharded Atomic| Strictly Monotonic | Very High   | Medium          |
|    Redis / Raft Counters | Per Channel (1..N) | (>500k ops) | (Key sharding)  |
+-------------------------------------------------------------------------------+

Discord’s Approach: Snowflake IDs with Deterministic Tie-Breaking

Discord generates 64-bit unique IDs using a modified Twitter Snowflake format:

+-------------------------------------------------------------------------------+
| 1 bit |  41 bits: Milliseconds since custom epoch  | 10 bits: Node | 12 bits: |
| Unused|  (Gives 69 years of monotonically sorted IDs) | Machine ID  | Sequence |
+-------------------------------------------------------------------------------+

1. Millisecond Sortability: The first 41 bits represent physical time at the generator node. Messages created later naturally sort higher.
2. Same-Millisecond Ordering: The trailing 12-bit sequence counter increments for every message generated by that machine within the exact same millisecond ($4,096\text{ IDs/ms/node}$).
3. Partition Keying: In storage, messages are clustered by (channel_id, message_id ASC). If two messages collide on the identical millisecond, the deterministic machine ID and sequence counter act as an absolute tie-breaker.

5. Distributed Ephemeral Presence: Heartbeat Leases

Presence informs users whether their contacts are Online, Away, or Offline. While seemingly simple, presence is notoriously difficult to scale:

• In a chat application with $10,000,000$ online users, if each user has $200$ friends, a naive broadcast of every status change generates $2,000,000,000$ presence fan-out updates per state transition!
• Mobile devices constantly experience silent disconnections (entering elevators, dead batteries, dropped carrier cells) where the client cannot send an explicit DISCONNECT frame.

Presence Heartbeat Lease Lifecycle:
---------------------------------------------------------------------------------
Client                     Gateway Node                      Redis Presence Store
  |                             |                                      |
  |--- WSS Ping (every 25s) --->|                                      |
  |                             |--- SET user:123:presence "online" -->|
  |                             |    EX 30                             |
  |                             |                                      |
  |<-- WSS Pong (ACK) ----------|                                      |
  |                             |                                      |
  |    ... 35 seconds pass ...  |                                      |
  |    (Client phone dies!)     |                                      |
  |    [No Ping received]       |                                      |
  |                             |                                      |
  |                             |                               [Lease Expires!]
  |                             |                               Key deleted.
  |                             |                               User transitions
  |                             |                               to OFFLINE.

The Ephemeral Lease Pattern

Rather than writing persistent state on connection, presence relies on ephemeral lease renewal:

1. Heartbeat Pings: Every client transmits an empty WebSocket ping frame every $25\text{ seconds}$.
2. Atomic TTL Lease: The receiving gateway updates an in-memory key in Redis with a 30-second Time-To-Live (TTL):

   SET user:alice:online "gateway_04" EX 30

3. Silent Drop Handling: If Alice loses signal, no ping arrives. After $30\text{ seconds}$, Redis automatically evicts the key via key expiration.
4. Presence Polling vs. Push Debouncing:
   • Instead of broadcasting presence to all $200$ friends instantaneously, presence changes are debounced. If a user flickers offline and online within $10\text{ seconds}$, no broadcast occurs.
   • For massive channels (e.g., Discord servers with $500,000$ members), presence is pulled on demand: only the visible users in the client’s current UI viewport ($50$ users) have their presence evaluated.

Quantitative Heartbeat Bandwidth & Redis Load

For $N=10,000,000$ concurrent users transmitting a WebSocket Ping every $T_{\text{hb}}=25\text{ seconds}$:

• Redis Lease Write Rate: $400,000\text{ SET ... EX 30 ops/sec}$, which shards effortlessly across $8$ Redis Cluster primaries ($\approx 50,000\text{ ops/sec/shard}$).
• Aggregate Wire Bandwidth: While the raw RFC 6455 Ping/Pong frame is only $2-6\text{ bytes}$, full Ethernet + IPv4 + TCP + TLS 1.3 record framing consumes $\approx 90\text{ bytes}$ per direction ($\approx 180\text{ bytes}$ round-trip): $$B_{\text{hb}}=400,000\text{ /s}\times 180\text{ bytes}\times 8\text{ bits/byte}=576\times 10^6\text{ bits/s}=0.576\text{ Gbps}$$ By comparison, $2\text{-second}$ HTTP/1.1 short polling ($5,000,000\text{ polls/s}\times 1,600\text{ B}$ round-trip headers) would burn $64\text{ Gbps}$—over $111\times$ more edge bandwidth before a single chat message is sent.

6. Storage Architecture: Dual-Tier Persistence

A robust chat storage tier must satisfy two conflicting access patterns:

1. Recent Message Ingestion & Reading: Extreme write throughput, followed immediately by high-frequency reads as channel participants consume the new messages.
2. Historical Search & Endless Scroll: Cold, immutable reads querying historical conversations dating back months or years.

Hot Tier: In-Memory Redis Timeline

Active channels maintain their recent $1,000$ messages in a Redis Sorted Set:

• Key: channel:<channel_id>:timeline
• Score: Monotonic sequence_id (or Snowflake ID)
• Member: Compressed JSON/Protobuf payload { id, sender, body, timestamp }

ZADD channel:dev-chat 1001 "Alice: Morning team"
ZADD channel:dev-chat 1002 "Bob: Standup in 5 mins"
ZADD channel:dev-chat 1003 "Charlie: Joining now"

-- Fetch the last 50 messages instantly:
ZREVRANGEBYSCORE channel:dev-chat +inf -inf LIMIT 0 50

Cold Tier: ScyllaDB / Cassandra Append-Only Wide-Column Store

Cassandra and ScyllaDB are purpose-built for chat storage: they use a Log-Structured Merge (LSM) Tree architecture (covered in CORE-M08-003) that converts all random writes into blazing-fast sequential disk appends with zero read-before-write overhead.

CREATE KEYSPACE chat_system 
WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east': 3, 'eu-west': 3};

CREATE TABLE chat_system.messages_by_channel (
    channel_id uuid,
    bucket_epoch int,       -- Partition bucket: year_month (e.g. 202610)
    sequence_id bigint,     -- Monotonically increasing sequence
    message_id timeuuid,    -- Unique message identifier
    sender_id uuid,
    content text,
    media_urls list<text>,
    PRIMARY KEY ((channel_id, bucket_epoch), sequence_id)
) WITH CLUSTERING ORDER BY (sequence_id ASC);

Why Composite Partition Key (channel_id, bucket_epoch)?

In Cassandra, all rows sharing a partition key reside on the same physical storage node. If a channel exists for 8 years and accumulates $20,000,000$ messages, a single partition key (channel_id) would grow to tens of gigabytes, creating a giant partition hotspot that destroys compaction performance.

By introducing bucket_epoch (e.g., year and month 202610), partitions are automatically sliced into predictable $50\text{ MB}-100\text{ MB}$ chunks distributed uniformly across the storage cluster.

7. Worked Example: 1:1 Direct Messaging vs. Massive Broadcast Fan-Out

Routing algorithms change dramatically depending on the cardinality of the conversation channel.

Scenario A: 1:1 Direct Message (Alice to Bob)

1. Alice sends { to: "bob", body: "Hello" } over her WebSocket connection to Gateway-1.
2. Gateway-1 queries the Redis Session Directory:

   HGET user:bob:sessions active_gateway
   --> Returns "Gateway-8"

3. Gateway-1 publishes the message directly to the Redis Pub/Sub topic for Gateway-8:

   PUBLISH gateway:gw-8:events '{"recipient":"bob", "body":"Hello", "seq":401}'

4. Gateway-8 receives the event over its internal subscriber loop, locates Bob’s active socket in its local socket map, and pushes the frame to Bob’s device.
5. Total hop latency: $8-18\text{ ms}$.

Scenario B: Massive Group Fan-Out (Discord Server with 200,000 Members)

If a celebrity or server admin posts a message in a channel with $200,000$ connected users:

• Catastrophic Failure Mode (Direct Fan-Out): The sender’s gateway attempts to duplicate the message $200,000$ times and send $200,000$ network payloads. The gateway node’s network interface controller (NIC) saturates, socket buffers explode, and the node crashes due to Out-Of-Memory (OOM).

Production Architecture for Massive Channels:

1. Single Publish to Ingestion Log: The gateway writes the message once to a distributed message log (Kafka / Pulsar) partitioned by channel_id.
2. Worker Cluster Fan-Out: A fleet of background consumer workers reads the message from the topic.
3. Gateway-Level Aggregation: Workers query the session store and group recipients by their connected gateway:
   • Gateway 1: 14,200 connected channel members.
   • Gateway 2: 18,900 connected channel members.
   • Gateway 3: 11,400 connected channel members.
4. Multicast to Gateways: The worker sends one single copy of the message to Gateway 1 with a recipient list.
5. Local Socket Multiplexing: Gateway 1 iterates through its local in-memory socket pointers, broadcasting the pre-serialized frame directly across the local sockets in user space.

8. Offline Synchronization: Watermark Cursor Reconciliation

When a user closes their laptop or drives through a tunnel, they disconnect. While offline, 42 new messages arrive in their team channels. How does the device catch up upon reconnection without duplicate delivery or dropped messages?

Watermark Cursor Reconciliation:
---------------------------------------------------------------------------------
Client Local Storage                     Storage Cluster (ScyllaDB / Redis)
--------------------                     ----------------------------------
Last Acknowledged Watermark:             Latest Channel Messages:
[channel_id: c_1, seq_id: 1040]          - Seq 1040: "See you tomorrow"
                                         - Seq 1041: "Wait, one more thing"
                                         - Seq 1042: "Check the PR"
                                         - Seq 1043: "LGTM, merged!"

1. Reconnect: Client sends handshake:
   SYNC { channel: "c_1", since_seq: 1040 }

2. Server executes range query:
   SELECT * FROM messages WHERE channel = 'c_1' AND sequence_id > 1040 ORDER BY sequence_id ASC

3. Server returns Delta Batch:
   [Seq 1041, Seq 1042, Seq 1043]

4. Client appends delta to local SQLite database and updates local watermark to 1043.

The Three Guarantees of Watermark Synchronization

1. Idempotence: If the client receives the delta batch, crashes before saving its watermark, and reconnects requesting since_seq: 1040 again, the server re-transmits the identical messages. Because every message has a deterministic sequence_id, the client’s local SQLite database executes INSERT OR IGNORE, preventing duplicate UI bubbles.
2. Zero Gap Detection: If the client receives Seq 1041 and Seq 1043 without receiving Seq 1042, the client immediately detects a missing sequence gap and triggers an explicit hole-repair request for sequence_id = 1042.
3. Multi-Device Coherence: Each physical device (laptop, tablet, phone) maintains its own independent watermark cursor. Reading a message on the laptop advances the server-side read_receipt_watermark, which pushes a silent background synchronization frame to the phone to clear its unread badge icon.

9. Operational Failure Modes & Production Troubleshooting

+-------------------------------------------------------------------------------+
|                      COMMON PRODUCTION FAILURE MODES                          |
+-------------------------------------------------------------------------------+
| Symptom                 | Root Cause               | Mitigation Strategy      |
+-------------------------+--------------------------+--------------------------+
| 1. Gateway Cluster OOM  | Slow client consumers    | Enforce strict socket    |
|    Collapse             | causing unbounded TCP    | write buffer high-water- |
|                         | write buffer growth      | marks; drop stale conn.  |
|                         |                          |                          |
| 2. Reconnection Thundering| Network glitch causes  | Randomized exponential   |
|    Herd                 | millions of clients to   | backoff + Jitter on all  |
|                         | reconnect simultaneously | client reconnection loops|
|                         |                          |                          |
| 3. Zombie TCP Sockets   | Silent mobile drops      | Aggressive TCP keepalives|
|    Draining FD Pool     | without FIN/RST packet   | + application-level ping |
|                         |                          | deadlines (30s timeout)  |
|                         |                          |                          |
| 4. Presence Redis Hot   | High-frequency presence  | Presence aggregation,    |
|    Key Saturation       | writes for popular bot   | debouncing, and viewport |
|                         | accounts / celebrities   | on-demand fetch only     |
+-------------------------------------------------------------------------------+

Deep Dive: The Slow Consumer Buffer Poisoning Vulnerability

When an online user experiences high packet loss (e.g. entering a parking garage), TCP reduces its congestion window (cwnd) to 1 segment. If active group chats continue streaming messages at $50\text{ messages/second}$, the gateway kernel cannot flush data onto the wire.

If the gateway application queues outgoing frames in user-space RAM without an upper bound:

1. Memory allocated to that single user’s socket grows from $18\text{ KB}$ to $50\text{ MB}$.
2. A few thousand slow clients across a gateway node consume $100\text{ GB}$ of RAM.
3. The Linux Out-of-Memory (OOM) Killer terminates the gateway process, severing $500,000$ innocent users!

Remediation: Implement a strict Socket Write Buffer Watermark. If an individual client’s pending write queue exceeds $1\text{ MB}$ or $500$ unacknowledged frames, the gateway forcefully terminates the socket with code 4408 (Slow Consumer). The client will reconnect cleanly later and synchronize via watermark cursors.

10. Concrete Implementation: WebSocket Gateway & Heartbeat Lease Engine

The following complete, production-grade TypeScript implementation demonstrates a high-concurrency WebSocket Gateway with connection management, ephemeral heartbeat leases, and message routing.

// gateway-server.ts
export interface ChatMessage {
  channelId: string;
  senderId: string;
  sequenceId: number;
  content: string;
  timestamp: number;
}

export interface ClientConnection {
  userId: string;
  gatewayId: string;
  lastHeartbeat: number;
  send: (payload: string) => void;
  close: () => void;
}

type EventListener = (payload: unknown) => void;

class GatewayEventBus {
  private readonly listeners = new Map<string, EventListener[]>();

  public on(event: string, listener: EventListener): void {
    const list = this.listeners.get(event) ?? [];
    list.push(listener);
    this.listeners.set(event, list);
  }

  protected emit(event: string, payload: unknown): void {
    for (const listener of this.listeners.get(event) ?? []) {
      listener(payload);
    }
  }
}

/**
 * In-memory Mock of Redis Session & Presence Registry
 */
export class DistributedSessionStore {
  private sessions = new Map<string, { gatewayId: string; expiresAt: number }>();

  public setPresenceLease(userId: string, gatewayId: string, ttlSeconds = 30): void {
    this.sessions.set(userId, {
      gatewayId,
      expiresAt: Date.now() + ttlSeconds * 1000,
    });
  }

  public getSession(userId: string): { gatewayId: string; online: boolean } | null {
    const session = this.sessions.get(userId);
    if (!session) return null;
    if (Date.now() > session.expiresAt) {
      this.sessions.delete(userId);
      return null;
    }
    return { gatewayId: session.gatewayId, online: true };
  }

  public removeSession(userId: string): void {
    this.sessions.delete(userId);
  }
}

/**
 * WebSocket Gateway Node responsible for managing stateful client connections
 */
export class WebSocketGatewayNode extends GatewayEventBus {
  public readonly gatewayId: string;
  private readonly sessionStore: DistributedSessionStore;
  private readonly localClients = new Map<string, ClientConnection>();
  private readonly sequenceGenerators = new Map<string, number>();
  private heartbeatIntervalTimer: ReturnType<typeof setInterval> | null = null;

  constructor(gatewayId: string, sessionStore: DistributedSessionStore) {
    super();
    this.gatewayId = gatewayId;
    this.sessionStore = sessionStore;
  }

  public start(heartbeatCheckIntervalMs = 5000): void {
    this.heartbeatIntervalTimer = setInterval(() => {
      this.reapZombieSockets();
    }, heartbeatCheckIntervalMs);
  }

  public stop(): void {
    if (this.heartbeatIntervalTimer !== null) {
      clearInterval(this.heartbeatIntervalTimer);
      this.heartbeatIntervalTimer = null;
    }
    for (const client of this.localClients.values()) {
      client.close();
    }
    this.localClients.clear();
  }

  public registerClient(userId: string, sendFn: (msg: string) => void, closeFn: () => void): void {
    const client: ClientConnection = {
      userId,
      gatewayId: this.gatewayId,
      lastHeartbeat: Date.now(),
      send: sendFn,
      close: closeFn,
    };

    this.localClients.set(userId, client);
    this.sessionStore.setPresenceLease(userId, this.gatewayId, 30);
    this.emit('client_connected', { userId, gatewayId: this.gatewayId });
  }

  public handleHeartbeat(userId: string): boolean {
    const client = this.localClients.get(userId);
    if (!client) return false;

    client.lastHeartbeat = Date.now();
    this.sessionStore.setPresenceLease(userId, this.gatewayId, 30);
    return true;
  }

  public sendMessage(channelId: string, senderId: string, text: string): ChatMessage {
    const currentSeq = (this.sequenceGenerators.get(channelId) || 0) + 1;
    this.sequenceGenerators.set(channelId, currentSeq);

    const message: ChatMessage = {
      channelId,
      senderId,
      sequenceId: currentSeq,
      content: text,
      timestamp: Date.now(),
    };

    this.emit('message_published', message);
    return message;
  }

  public deliverLocally(recipientId: string, message: ChatMessage): boolean {
    const client = this.localClients.get(recipientId);
    if (!client) return false;

    client.send(JSON.stringify({
      type: 'MESSAGE_DELIVERY',
      data: message,
    }));
    return true;
  }

  public disconnectClient(userId: string): void {
    const client = this.localClients.get(userId);
    if (client) {
      client.close();
      this.localClients.delete(userId);
      this.sessionStore.removeSession(userId);
      this.emit('client_disconnected', { userId, gatewayId: this.gatewayId });
    }
  }

  private reapZombieSockets(): void {
    const now = Date.now();
    const timeoutMs = 30000;

    for (const [userId, client] of this.localClients.entries()) {
      if (now - client.lastHeartbeat > timeoutMs) {
        this.disconnectClient(userId);
      }
    }
  }

  public getConnectedClientCount(): number {
    return this.localClients.size;
  }
}

function runDemo(): void {
  console.log("=== Real-Time Chat & Distributed WebSocket Gateway Architecture ===");
  const sessionStore = new DistributedSessionStore();
  const gateway = new WebSocketGatewayNode("gw-us-east-1", sessionStore);
  console.log("Initialized WebSocket Gateway on 'gw-us-east-1'");
  gateway.registerClient("user-alice", (msg) => console.log(`  [Alice Received]: ${msg}`), () => {});
  gateway.registerClient("user-bob", (msg) => console.log(`  [Bob Received]: ${msg}`), () => {});
  console.log(`Connected clients count: ${gateway.getConnectedClientCount()}, Bob online=${sessionStore.getSession("user-bob")?.online}`);
  const chatMsg = gateway.sendMessage("chan-general", "user-alice", "Hey Bob, check the Raft log!");
  console.log(`Dispatched chat message seq=${chatMsg.sequenceId} in '${chatMsg.channelId}' across gateway cluster`);
  gateway.deliverLocally("user-bob", chatMsg);
  gateway.handleHeartbeat("user-bob");
}

runDemo();

11. Exercises with Observable Deliverables

Exercise 1: Calculate Global Cluster Socket Memory Footprint

Scenario: You are designing the edge gateway tier for an enterprise chat application with $25,000,000$ registered accounts and $4,000,000$ peak concurrent online users.

1. Given an average socket memory footprint of $16\text{ KB}$ (TCP buffers + TLS context + user session), calculate the total RAM needed across the gateway tier.
2. If your engineering standard mandates a maximum of $600,000$ connections per gateway node to allow for $50\%$ failover burst capacity, how many $32\text{ GB}$ nodes must you provision?

Exercise 2: Design the Offline Reconciliation Algorithm

Deliverable: Write a TypeScript or Python function reconcileChannelHistory(clientWatermark: number, channelId: string) that:

1. Queries the hot Redis Sorted Set for sequence IDs greater than clientWatermark.
2. Detects if there is a cache miss (i.e., if clientWatermark is older than the oldest message in Redis).
3. Seamlessly falls back to a ScyllaDB range query if a cache miss occurs, returning an in-order delta array to the client.

12. Summary & The Distributed Architect’s Rules of Thumb

+-------------------------------------------------------------------------------+
|                    THE CHAT ARCHITECT'S DECISION GUIDE                        |
+-------------------------------------------------------------------------------+
| Requirement               | Recommended Architecture                          |
+---------------------------+---------------------------------------------------+
| Connection Protocol       | WebSockets over TLS (WSS) with HTTP/2 fallback    |
| Edge Concurrency Model    | Async event loops (epoll/kqueue) in Go/Rust/C++   |
| Ephemeral Presence        | Heartbeat leases in Redis with 25s ping / 30s TTL |
| Direct Message Routing    | In-memory session directory + targeted Redis Pub/Sub|
| High-Volume Channels      | Async worker queue + Gateway-level multicast      |
| Message Ordering          | Per-channel monotonic sequence IDs / Snowflake    |
| Storage Tiering           | Redis Sorted Sets (Hot) + ScyllaDB LSM (Cold)     |
| Offline Catch-Up          | Watermark sequence cursors + gap hole-repair      |
+-------------------------------------------------------------------------------+

1. Decouple Gateways from Business Logic: Never execute expensive database transactions or heavy business logic on the WebSocket gateway nodes. The gateway’s sole duty is pumping bytes between OS sockets and the internal message broker.
2. Never Order Messages by Device Clock: Assume all client clocks are untrusted and out of sync. Always order messages using per-channel monotonic sequencers generated by server-side brokers or Raft logs.
3. Guard Against Slow Consumers Aggressively: One slow client must never be allowed to exhaust server memory. Set strict socket write buffer watermarks and drop connections that lag behind.
4. Debounce Presence Fan-Out: Never broadcast raw presence changes to hundreds of thousands of users simultaneously. Use heartbeats with TTL leases, debounce status flickers, and load presence on-demand for large server rosters.

13. Authoritative References & Version Context

• IETF RFC 6455 — The WebSocket Protocol: RFC 6455 (frame header wire format, Ping/Pong control opcodes 0x9/0xA, masking rules, and close status codes).
• Bo Ingram (Discord Engineering, 2023): How Discord Stores Trillions of Messages (composite (channel_id, bucket) partitioning, request coalescing data services, and migration from Apache Cassandra to ScyllaDB).
• Slack Engineering: Flannel: An Application-Level Edge Cache to Make Slack Scale (edge WebSocket termination, lazy viewport-driven presence/metadata hydration, and broadcast storm elimination).
• ScyllaDB Architecture & Data Modeling Documentation: Time-Series & Wide-Column Partitioning (baseline stack context: ScyllaDB 6.x / Apache Cassandra 5.0, Redis 7.2+, Apache Kafka 3.7+).