Apache Kafka revolutionized distributed stream processing by modeling messaging as a distributed append-only commit log. However, in Kafka, compute and storage are tightly coupled on the same broker: a broker serves producers and consumers while storing partition log segments on its local NVMe drive. When a partition grows to tens of terabytes or when a broker fails, Kafka must rebalance entire partitions across the network, causing massive I/O saturation and operational pain.
Apache Pulsar resolved this architectural coupling through a radical two-layer cloud-native architecture:
- Stateless Serving Layer (Pulsar Brokers): Handle client TCP connections, authorization, protocol handling, and caching. Because brokers hold zero persistent state, scaling up or down takes seconds with zero data rebalancing.
- Distributed Log Storage Layer (Apache BookKeeper): A specialized, low-latency distributed append-only storage engine. Data is organized into immutable Ledgers distributed across a pool of storage nodes called Bookies.
- Tiered Storage Offload: Seamlessly offloads historical ledger segments to cheap cloud object storage (Amazon S3 / Google Cloud Storage) while keeping them transparently queryable through standard consumer APIs.
This module details the protocols, consensus quorums, and disk I/O architectures that make Pulsar and BookKeeper a formidable enterprise alternative to traditional messaging systems.