Apache ZooKeeper & Ratis: Consensus, Znodes and Coordination Primitives

1. Context & Architecture Reality

Project Identity, Version & License Baseline

SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED SYSTEMS

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Phase 2: Synchronization (Log Alignment) & Phase 3: Atomic Broadcast (2-Phase Commit)

Establishes the ingress and structural boundary across Phase 2: Synchronization (Log Alignment) & Phase 3: Atomic Broadcast (2-Phase Commit): Followers sync missing proposals or truncate uncommitted epoch logs to match the newly elected Leader's history; Client Write → Leader: PROPOSE → Followers: ACK → Leader: COMMIT Simple 2PC without abort: Once proposed, transaction must commit!.

Architectural Invariant: Client Write → Leader: PROPOSE → Followers: ACK → Leader: COMMIT Simple 2PC without abort: Once proposed, transaction must commit!.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. lock-0001 & lock-0002

Coordinates core data-plane execution and state transitions through lock-0001 & lock-0002: Holds the lock!; Watches lock-0001; Watches 0002.

Architectural Invariant: Client i sets a Watch ONLY on node (i - 1). When Client 1 releases its lock, only Client 2 is notified! Prevents 10,000 clients from stampeding ZooKeeper.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. / (Root) & Embeddable Open-Source Raft Engine

Enforces durability, state convergence, and output verification across / (Root) & Embeddable Open-Source Raft Engine: &#9500;&#9472;&#9472; /services (Persistent) &#9500;&#9472;&#9472; /services/web-01 (Ephemeral: auto-deletes on session loss!) &#9500;&#9472;&#9472; /locks (Persistent) &#9500;&#9472;&#9472; /locks/lock-0000000001 (Ephemeral Sequential) &#9492;&#9472;&#9472; /locks/lock-0000000002 (Ephemeral Sequential) &#9492;&#9472;&#9472; /config (Persistent, payload < 1MB); Znode memory limit: 1 MB. Entire tree fits completely inside RAM!; Decouples Raft consensus from storage: Implements StateMachine interface.

Architectural Invariant: Linearizable reads via Raft ReadIndex leases (Zero log disk writes!).

In a distributed system spanning hundreds of machines, solving classic coordination challenges is notoriously difficult:

• Leader Election: How do 50 worker nodes agree on which single node is the active master without risking split-brain dual leaders?
• Distributed Mutex Locking: How do concurrent worker processes serialize access to a shared resource across the network?
• Dynamic Cluster Membership: How do cluster components immediately detect when a node crashes or disconnects without polling?

In 2008, Yahoo! Research developed and open-sourced Apache ZooKeeper, an implementation of an atomic broadcast coordination service modeled after Google’s proprietary Chubby lock service. ZooKeeper became the indispensable coordination foundation for the entire big data generation, serving as the master coordinator for Apache Hadoop, Apache Kafka, Apache HBase, Apache Solr, and Apache Storm.

ZooKeeper provides a familiar hierarchical namespace modeled after a Unix file system, called the Znode Tree. However, unlike a standard file system:

• Every znode can hold binary metadata (up to $1\text{ MB}$).
• The entire state tree is stored in-memory in JVM heap, replicated across an ensemble of 3 or 5 servers running the Zab (ZooKeeper Atomic Broadcast) consensus protocol.
• Clients can register Watchers on znodes to receive immediate, asynchronous push notifications when data changes or nodes disconnect.

+--------------------------------------------------------------------------------------------------+
|                            ZOOKEEPER ENSEMBLE AND ZAB CONSENSUS                                  |
|                                                                                                  |
|   Client Applications                                                                            |
|   (Kafka Brokers / HBase RegionServers / Distributed Workers)                                    |
|        |                                                                                         |
|        v TCP Connection (Session Leases & Watchers)                                              |
|   +-----------------------------------------------------------------------------------------+    |
|   | ZOOKEEPER 3-NODE / 5-NODE ENSEMBLE                                                      |    |
|   |                                                                                         |    |
|   |         +-----------------------+                                                       |    |
|   |         |   LEADER (Node 1)     | <--- Receives all Writes (Propose -> Commit)          |    |
|   |         |   Zxid: 0x200000004   |                                                       |    |
|   |         +-----------+-----------+                                                       |    |
|   |                    / \                                                                  |    |
|   |     Zab Protocol  /   \  Quorum Ack (Q = 2 of 3)                                        |    |
|   |                  v     v                                                                |    |
|   |   +-------------------+   +-------------------+                                         |    |
|   |   | FOLLOWER (Node 2) |   | FOLLOWER (Node 3) | <--- Serves Local Reads                 |    |
|   |   | Local In-Memory DB|   | Local In-Memory DB|                                         |    |
|   |   +-------------------+   +-------------------+                                         |    |
|   +-----------------------------------------------------------------------------------------+    |
+--------------------------------------------------------------------------------------------------+

2. Core Problem & Failure Modes

Distributed coordination systems are high-leverage components: if the coordination service stalls, the entire platform grinds to a halt.

1. The Herd Effect Stampede: A naive distributed lock creates a shared znode /mutex. 5,000 worker threads set a Watcher on /mutex waiting for it to be deleted. When the lock holder finishes and deletes /mutex, ZooKeeper fires 5,000 asynchronous watch notifications across the network simultaneously. All 5,000 clients wake up, hammer ZooKeeper with create() requests, and exhaust network socket buffers, freezing the cluster.

2. JVM Garbage Collection Session Eviction: A client process running in Java experiences a 30-second “Stop-the-World” Garbage Collection pause. During the pause, the client cannot send its periodic heartbeat ping (tickTime). ZooKeeper marks the client session expired and deletes all its Ephemeral Znodes (including its active master leader lease). When the JVM wakes up, it continues acting as leader while another node has already been elected, causing split-brain execution!

3. Disk Snapshot Stalls & Jute Buffer Overflows: ZooKeeper periodically takes a fuzzy snapshot of its entire in-memory datatree to disk while writing transaction logs. If the snapshot disk shares I/O channels with the transaction log (WAL), disk write latency spikes from $1\text{ ms}$ to $1,500\text{ ms}$, causing heartbeats between ensemble peers to time out and triggering unnecessary leader elections.

3. Technical Deep Dive

The Zab Protocol Lifecycle

Zab is a crash-recovery, primary-backup atomic broadcast protocol specifically optimized for hierarchical tree state machines. Zab operates through two alternating modes: Crash Recovery and Atomic Broadcast.

+--------------------------------------------------------------------------------------------------+
| PHASE 1: LEADER ELECTION (Discovery)                                                             |
| - Nodes vote using Fast Leader Election.                                                         |
| - A candidate wins if supported by a quorum (N/2 + 1) and holds the highest Zxid.               |
| - New Leader increments the Epoch ID: epoch_new = epoch_max + 1                                 |
+--------------------------------------------------------------------------------------------------+
                                               |
                                               v
+--------------------------------------------------------------------------------------------------+
| PHASE 2: SYNCHRONIZATION                                                                         |
| - Followers connect to the new Leader.                                                           |
| - Leader compares follower Zxid with its own commit history.                                    |
| - Missing proposals are streamed (DIFF); uncommitted proposals from old dead epochs are         |
|   truncated (TRUNC).                                                                             |
| - Leader is officially established once a quorum of followers are in sync.                      |
+--------------------------------------------------------------------------------------------------+
                                               |
                                               v
+--------------------------------------------------------------------------------------------------+
| PHASE 3: ATOMIC BROADCAST (Normal Operation)                                                     |
| - All writes funnel through the Leader.                                                          |
| - Step 1: Leader assigns monotonically increasing Zxid and sends PROPOSAL to followers.         |
| - Step 2: Followers append to WAL disk and respond with ACK.                                     |
| - Step 3: Once Leader receives Quorum ACKs (e.g., 2 of 3), it broadcasts COMMIT.                |
| - Step 4: Leader and followers apply mutation to in-memory DataTree.                             |
+--------------------------------------------------------------------------------------------------+

The 64-Bit Zxid Structure

Every transaction in ZooKeeper is uniquely identified by a 64-bit integer called a zxid:

• Epoch (Upper 32 bits): Incremented by 1 during every successful leader election. No two leaders can ever share the same epoch.
• Counter (Lower 32 bits): Sequential counter starting at 0 for every new epoch.

Causal Invariant: If $zxi{d}_A<zxi{d}_B$, then transaction $A$ was proposed strictly before transaction $B$ in real causal time, ensuring total order across all followers.

The Herd-Effect-Free Distributed Lock Algorithm

To acquire a lock without triggering a herd stampede:

1. Every client creates an Ephemeral Sequential Znode under the lock directory: $$\text{Path}=\text{create}("/\text{locks/lock-}",\text{EPHEMERAL\_SEQUENTIAL})$$
2. The client calls getChildren("/locks") and inspects the assigned sequence numbers (e.g., lock-0000000001, lock-0000000002…).
3. If the client’s created path has the lowest sequence number, the client holds the lock!
4. If not, the client locates the znode in the list with the immediate predecessor sequence number and registers a Watcher strictly on that predecessor node:

[/locks/lock-0001] <--- Holds Lock (Client 1)
        ^
        | (Watched by Client 2 ONLY)
[/locks/lock-0002] <--- Waiting (Client 2)
        ^
        | (Watched by Client 3 ONLY)
[/locks/lock-0003] <--- Waiting (Client 3)

When Client 1 completes and deletes lock-0001, ZooKeeper fires a watch event to exactly one recipient (Client 2). Client 2 wakes up and acquires the lock. All other 4,998 waiting clients remain asleep, eliminating the herd effect completely!

4. System Design Patterns & Trade-offs

+----------------------------------------------------------------------------------------------------+
| TRADEOFF MATRIX: DISTRIBUTED COORDINATION TECHNOLOGIES                                             |
+--------------------------+-----------------------+-----------------------+-------------------------+
| Feature                  | Apache ZooKeeper      | etcd (CoreOS)         | Apache Ratis            |
+--------------------------+-----------------------+-----------------------+-------------------------+
| Consensus Protocol       | Zab (Paxos-variant)   | Raft Consensus        | Pure Raft (Embeddable)  |
| Implementation Language  | Java (JVM Heap)       | Go (B-Tree + WAL)     | Java (Embeddable lib)   |
| Deployment Model         | Standalone Ensemble   | Standalone Cluster    | Embedded in application |
| Client Watcher Semantics | One-time trigger (old)| Continuous streams    | State machine hooks     |
| Memory Footprint         | Entire state in RAM   | On-disk B-Tree + Page | User-defined storage    |
+--------------------------+-----------------------+-----------------------+-------------------------+

5. Practical Implementation

Below is a complete TypeScript simulation demonstrating ZooKeeper’s core coordination primitives:

1. Hierarchical Znode tree with Persistent and Ephemeral nodes.
2. Asynchronous Watcher notifications.
3. Herd-Effect-Free Distributed Lock Manager using ephemeral sequential nodes.

export type ZnodeType = 'PERSISTENT' | 'EPHEMERAL';

export interface Znode {
  path: string;
  data: string;
  type: ZnodeType;
  ownerSessionId?: string;
  version: number;
  children: Map<string, Znode>;
}

export type WatcherCallback = (event: { path: string; type: 'DELETED' | 'DATA_CHANGED' }) => void;

/**
 * In-Memory ZooKeeper Replicated DataTree Simulator
 */
export class ZooKeeperDataTree {
  private root: Znode = {
    path: '/',
    data: '',
    type: 'PERSISTENT',
    version: 0,
    children: new Map()
  };

  private watchers: Map<string, WatcherCallback[]> = new Map();
  private sequenceCounter: number = 0;

  /**
   * Creates a znode with support for EPHEMERAL and SEQUENTIAL flags
   */
  public create(
    path: string,
    data: string,
    type: ZnodeType,
    isSequential: boolean = false,
    sessionId?: string
  ): string {
    let finalPath = path;
    if (isSequential) {
      const seqStr = (++this.sequenceCounter).toString().padStart(10, '0');
      finalPath = `${path}-${seqStr}`;
    }

    const segments = finalPath.split('/').filter(Boolean);
    let current = this.root;

    for (let i = 0; i < segments.length - 1; i++) {
      const seg = segments[i];
      if (!current.children.has(seg)) {
        throw new Error(`NodeDoesNotExist: parent /${segments.slice(0, i + 1).join('/')} missing`);
      }
      current = current.children.get(seg)!;
    }

    const leafName = segments[segments.length - 1];
    if (current.children.has(leafName)) {
      throw new Error(`NodeExists: ${finalPath}`);
    }

    const newNode: Znode = {
      path: finalPath,
      data,
      type,
      ownerSessionId: type === 'EPHEMERAL' ? sessionId : undefined,
      version: 0,
      children: new Map()
    };

    current.children.set(leafName, newNode);
    return finalPath;
  }

  /**
   * Registers a one-time Watcher on a znode
   */
  public watch(path: string, callback: WatcherCallback): void {
    if (!this.watchers.has(path)) {
      this.watchers.set(path, []);
    }
    this.watchers.get(path)!.push(callback);
  }

  /**
   * Deletes a znode and fires registered one-time watchers
   */
  public delete(path: string): boolean {
    const segments = path.split('/').filter(Boolean);
    let current = this.root;

    for (let i = 0; i < segments.length - 1; i++) {
      if (!current.children.has(segments[i])) return false;
      current = current.children.get(segments[i])!;
    }

    const leafName = segments[segments.length - 1];
    if (!current.children.has(leafName)) return false;

    current.children.delete(leafName);

    // Fire and clear one-time watchers
    const callbacks = this.watchers.get(path) || [];
    this.watchers.delete(path);
    for (const cb of callbacks) {
      cb({ path, type: 'DELETED' });
    }

    return true;
  }

  public getChildren(parentPath: string): string[] {
    const segments = parentPath.split('/').filter(Boolean);
    let current = this.root;

    for (const seg of segments) {
      if (!current.children.has(seg)) return [];
      current = current.children.get(seg)!;
    }

    return Array.from(current.children.keys()).sort();
  }
}

/**
 * Herd-Free Distributed Mutex Lock Client
 */
export class DistributedLockClient {
  private currentLockPath?: string;

  constructor(
    private readonly zk: ZooKeeperDataTree,
    private readonly sessionId: string,
    private readonly lockDir: string = '/locks'
  ) {}

  public async acquireLock(): Promise<boolean> {
    // Step 1: Create Ephemeral Sequential Znode
    this.currentLockPath = this.zk.create(
      `${this.lockDir}/lock`,
      this.sessionId,
      'EPHEMERAL',
      true,
      this.sessionId
    );

    return new Promise((resolve) => {
      this.checkAndWatch(resolve);
    });
  }

  private checkAndWatch(onAcquired: (val: boolean) => void): void {
    const children = this.zk.getChildren(this.lockDir);
    const myLeaf = this.currentLockPath!.split('/').pop()!;
    const myIndex = children.indexOf(myLeaf);

    if (myIndex === 0) {
      // Lowest sequence number holds the lock!
      onAcquired(true);
      return;
    }

    // Step 2: Eliminate Herd Effect! Watch ONLY the immediate predecessor
    const predecessorLeaf = children[myIndex - 1];
    const predecessorPath = `${this.lockDir}/${predecessorLeaf}`;

    this.zk.watch(predecessorPath, (event) => {
      if (event.type === 'DELETED') {
        // Predecessor released or crashed; re-check lock standing
        this.checkAndWatch(onAcquired);
      }
    });
  }

  public releaseLock(): void {
    if (this.currentLockPath) {
      this.zk.delete(this.currentLockPath);
      this.currentLockPath = undefined;
    }
  }
}

async function runDemo(): Promise<void> {
  console.log("=== Apache ZooKeeper Architecture: Data Tree & Distributed Locks ===");
  const zk = new ZooKeeperDataTree();
  zk.create("/services", "registry", "PERSISTENT");
  zk.create("/services/auth", "node-1:8080", "EPHEMERAL", false, "session-auth-1");
  console.log("[1] Created persistent node /services and ephemeral node /services/auth");

  // Initialize /locks root for Herd-Free Distributed Lock contention
  zk.create("/locks", "mutex-root", "PERSISTENT");
  const clientA = new DistributedLockClient(zk, "client-A", "/locks");
  const clientB = new DistributedLockClient(zk, "client-B", "/locks");

  const acquiredA = await clientA.acquireLock();
  console.log(`[2] Client A acquired lock: ${acquiredA} (active znodes: [${zk.getChildren("/locks").join(", ")}])`);

  // Client B contends for the lock; sets one-time Watcher strictly on Client A's predecessor znode
  let clientBAcquired = false;
  const lockBPromise = clientB.acquireLock().then((res) => {
    clientBAcquired = res;
    console.log(`[4] Predecessor watch fired -> Client B acquired lock: ${res} (active znodes: [${zk.getChildren("/locks").join(", ")}])`);
  });
  console.log(`[3] Client B queued behind predecessor (active znodes: [${zk.getChildren("/locks").join(", ")}], clientBAcquired=${clientBAcquired})`);

  // Client A releases lock -> wakes ONLY Client B
  clientA.releaseLock();
  await lockBPromise;
  clientB.releaseLock();
}

void runDemo();

Simulation Execution Output & State Transitions

=== Apache ZooKeeper Architecture: Data Tree & Distributed Locks ===
[1] Created persistent node /services and ephemeral node /services/auth
[2] Client A acquired lock: true (active znodes: [lock-0000000001])
[3] Client B queued behind predecessor (active znodes: [lock-0000000001, lock-0000000002], clientBAcquired=false)
[4] Predecessor watch fired -> Client B acquired lock: true (active znodes: [lock-0000000002])

6. Failure Scenarios & Edge Cases

1. The Asynchronous Watcher Window Race

In ZooKeeper, watchers are one-time triggers. When Client A receives a watch notification that /config changed, the watcher is consumed. If Client A issues getData("/config", watch=true) to re-arm the watcher, a second modification can occur in the tiny window between when the event fired and when the new watch was registered!

• Mitigation: Clients must read the znode data immediately upon receiving the event and inspect the monotonic version number. If the version is higher than expected, the client knows it missed an intermediate update and handles it gracefully.

2. Ephemeral Znode Disappearance on Brief Network Blips

If a client experiences network packet loss exceeding sessionTimeout (typically 10–20 seconds), ZooKeeper evicts the session and deletes all ephemeral znodes. If the client reconnects, its lock is gone, but the application code may continue executing assuming it still holds mutual exclusion.

• Mitigation: Implement Fencing Tokens. Every lock acquisition must return a monotonically increasing transaction counter. Storage layers reject any mutation carrying a stale fencing token.

7. Production Operations & Observability

ZooKeeper Four-Letter Word Monitoring Commands

Send 4-letter ASCII commands over raw TCP port 2181 using nc or echo:

• echo mntr | nc localhost 2181: Emits detailed operational metrics (leader/follower role, open client connections, average latency, outstanding proposals).
• echo ruok | nc localhost 2181: Returns imok if healthy.
• echo srvr | nc localhost 2181: Shows server version, zxid, and mode.

8. Security & Compliance

1. Kerberos / SASL Authentication: Secure production ensembles require Kerberos tickets for inter-ensemble and client-to-server connections, preventing unauthorized processes from spoofing cluster identity.
2. Granular ACL Permissions: Every znode supports access control lists (cdrwa: Create, Delete, Read, Write, Admin). System configurations (/config) should grant write permissions strictly to CI/CD deployment service accounts.

9. Real-World Case Studies

Apache Kafka’s 3-Year Quest to Eliminate ZooKeeper (KIP-500 / KRaft)

For over 10 years, Apache Kafka relied on ZooKeeper to store partition metadata, broker lists, and topic configurations.

• The Problem: When clusters grew to millions of partitions, ZooKeeper became the primary bottleneck. Loading metadata on startup took minutes, and watch notifications overloaded brokers.
• The Solution: Kafka introduced KRaft (Kafka Raft), embedding a specialized Raft quorum directly inside Kafka brokers. This eliminated ZooKeeper entirely, reducing partition failover times from minutes to milliseconds.

10. Technology Comparison Matrix

+----------------------------------------------------------------------------------------------------+
| DISTRIBUTED COORDINATION COMPARISON                                                                |
+----------------------+--------------------+--------------------+-----------------------------------+
| System               | Consensus Engine   | Storage Model      | Watcher Model                     |
+----------------------+--------------------+--------------------+-----------------------------------+
| Apache ZooKeeper     | Zab                | In-Memory DataTree | One-time asynchronous triggers    |
+----------------------+--------------------+--------------------+-----------------------------------+
| etcd                 | Raft               | B+Tree on BoltDB   | Continuous streaming gRPC watches |
+----------------------+--------------------+--------------------+-----------------------------------+
| HashiCorp Consul     | Raft               | In-Memory State    | Long-polling HTTP / Blocking query|
+----------------------+--------------------+--------------------+-----------------------------------+
| Apache Ratis         | Raft (Embeddable)  | Pluggable Java lib | Application callback interfaces   |
+----------------------+--------------------+--------------------+-----------------------------------+

11. Design Challenge & Exercise

Scenario: Active-Active Leader Election with Leases

You are designing the leader election coordinator for a distributed scheduler running across 3 AWS Availability Zones. Only one scheduler may be active at any second; if the active scheduler dies, a standby must take over within 5 seconds.

Requirements:

1. Formulate the exact znode path structure and ACL permissions.
2. Detail the sequence of events when the active leader experiences a 40-second network partition.
3. Design a fencing token mechanism so that database writes issued by a zombie leader are rejected with $100\%$ mathematical certainty.

12. Assessment

Review the 4 scenario-based quiz questions at the top of this lesson to test your mastery of Zab consensus, hierarchical znode models, and herd-effect-free distributed locking algorithms.

13. Summary & Architectural Decision Guide

• ZAB Protocol Epoch Ordering: 64-bit zxid ensures recovery from leader crashes without violating committed state.
• Hierarchical Tree of Znodes: Combines data storage with ephemeral presence and event watches.
• Sequential Watch Recipe: Solves thundering herds in distributed locks by watching only predecessor nodes.
• Embedded Consensus with Ratis: Modern big-data engines (Apache Ozone) embed Ratis Raft rather than deploying external ZooKeeper ensembles.

14. Authoritative References & Version Context

• Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed (2010), “ZooKeeper: Wait-free Coordination for Internet-scale Systems”, USENIX ATC ‘10 — usenix.org/conference/usenix-atc-10/zookeeper-wait-free-coordination-internet-scale-systems
• Flavio P. Junqueira, Benjamin C. Reed, and Marco Serafini (2011), “Zab: High-performance broadcast for primary-backup systems”, IEEE DSN ‘11 — doi.org/10.1109/DSN.2011.5958223
• Apache Ratis Project Documentation (ASF v3.0+) — ratis.apache.org
• Version Context: Architecture covers Apache ZooKeeper 3.9+ (dynamic reconfig), ZAB 1.0, and Apache Ratis 3.0+ embedded consensus.