When a single database instance exhausts vertical scaling limits—hitting storage capacity ceilings, connection pool saturation, or disk I/O bottlenecks—systems must scale horizontally by sharding (partitioning) data across multiple independent database nodes.

Sharding fundamentally changes how applications interact with data: single-node relational guarantees, foreign keys, secondary indexes, and atomic ACID transactions across multiple tables dissolve. This module deconstructs distributed database engineering: selecting optimal shard keys, range versus hash partitioning, routing tiers (Vitess, Citus), cross-shard scatter-gather query execution, distributed transaction coordination (Two-Phase Commit vs. Sagas), and non-blocking resharding migrations. You will evaluate how compound partition keys prevent hot-spot skew, why coordinator crashes during the 2PC prepare phase block participant row locks indefinitely, and how Dual-Write plus Change-Data-Capture (CDC) pipelines split live production shards with zero downtime.

What you will learn

  • Design horizontal sharding strategies balancing query routing efficiency against hot partition skew
  • Compare Range-based partitioning with Hash-based partitioning and consistent hashing topologies
  • Deconstruct Two-Phase Commit (2PC) protocol mechanics and prove coordinator blocking failure modes
  • Implement asynchronous Saga pattern orchestrators using compensating transactions and idempotency keys
  • Architect cross-shard scatter-gather query engines with distributed limit, sort, and aggregation operators
STAGE TOPICS

Topics in this chapter