When a dataset expands beyond the storage capacity of a single physical disk, or when read/write query throughput exceeds the I/O capacity of a single server, partitioning (sharding) becomes mandatory.
Partitioning divides a large database into smaller, independent subsets called partitions or shards. Each partition is assigned to a specific node, allowing the system to scale storage and throughput linearly by adding machines.
However, partitioning introduces complex distributed routing challenges:
- Hotspots and Skew: Why naive range partitioning creates massive write bottlenecks during timestamp-ordered workloads.
- Secondary Indexing: The fundamental architectural choice between Document-Partitioned (scatter-gather) and Term-Partitioned (global write coordination) secondary indexes.
- Rebalancing: How to migrate gigabytes of data to newly added nodes without dropping client connections or locking database tables.