1. Context & Architecture Reality
Project Identity, Version & License Baseline
| Project Component | Canonical Repository | Pinned Release | Governance / License | Edition Boundary |
|---|---|---|---|---|
| Vitess | vitessio/vitess | v19.0 (LTS) | CNCF Graduated (Apache 2.0) | Open source |
| ClickHouse | ClickHouse/ClickHouse | v24.3 (LTS) | ClickHouse Inc (Apache 2.0) | Open source |
1. Shard 1: Keyspace Range [-80) & Shard 2: Keyspace Range [80-)
Establishes the ingress and structural boundary across Shard 1: Keyspace Range [-80) & Shard 2: Keyspace Range [80-): VTTablet → MySQL Primary + Replicas Connection pooling, query killer, row cache; Streams binlog mutations for zero-downtime resharding and split cutovers in < 1s!; Index granularity = 8192 rows 1 index entry per mark block Entire index fits in RAM! Zero random disk seeks.
2. Row Store (MySQL / Postgres) & Column Store (ClickHouse)
Coordinates core data-plane execution and state transitions through Row Store (MySQL / Postgres) & Column Store (ClickHouse): [Row1: id, name, age, bio, addr]; [Row2: id, name, age, bio, addr]; Col Age: [25, 34, 19, 42, 55...].
3. ClickHouse: Vectorized Columnar MergeTree & Physical Data Layout: Row Store vs Column Store
Enforces durability, state convergence, and output verification across ClickHouse: Vectorized Columnar MergeTree, Physical Data Layout: Row Store vs Column Store, OLTP vs OLAP Architectural Tradeoffs within the Large-Scale Storage Architecture: Vitess Sharding and ClickHouse OLAP architecture.
At moderate scale (tens of gigabytes, a few thousand queries per second), a single well-tuned relational database server running PostgreSQL or MySQL handles application traffic easily. However, when an enterprise scales to tens of terabytes, hundreds of millions of users, or billions of events per day, single-node architectures reach a hard physical limit.
At hyper-scale, data workloads bifurcate into two diametrically opposed access patterns:
- Online Transaction Processing (OLTP): High-concurrency, low-latency, point reads and writes on individual user entities (e.g., user profiles, shopping carts, checkout payments). Requirements: ACID transactions, row-level locking, sub-10ms response times.
- Online Analytical Processing (OLAP): High-throughput aggregations, trend analysis, and reporting across billions of events (e.g., telemetry, log analytics, security audit trails, BI dashboards). Requirements: Scanning billions of rows per second, complex
GROUP BYandAVGoperations.
No single database engine can optimize both patterns simultaneously. Attempting to run real-time analytical queries on a transactional MySQL primary locks tables, exhausts memory buffers, and tanks user checkout transactions.
This architecture study deconstructs the two premier open-source solutions designed for each extreme:
- Vitess: The horizontal scaling engine created by YouTube to shard and manage thousands of MySQL nodes transparently.
- ClickHouse: The ultra-fast columnar analytical engine designed to aggregate hundreds of millions of rows per second per server.
+--------------------------------------------------------------------------------------------------+
| THE TWO POLES OF LARGE-SCALE STORAGE |
| |
| OLTP TRANSACTIONAL WORKLOAD (Vitess) OLAP ANALYTICAL WORKLOAD (ClickHouse) |
| | |
| - Workload: Single-row INSERT / UPDATE | - Workload: Batched bulk appends, wide scans|
| - Concurrency: 100,000 QPS | - Concurrency: 50 - 500 QPS |
| - Query: SELECT * FROM users WHERE id = 4821; | - Query: SELECT avg(latency) FROM logs |
| - Latency SLA: < 10ms | WHERE status = 500 GROUP BY country; |
| - Storage Engine: Row-Store (MySQL InnoDB) | - Latency SLA: 200ms - 2s (across 10B rows!)|
| - Architecture: Transparent Sharding via VTGate | - Storage Engine: Columnar (MergeTree) |
+--------------------------------------------------------------------------------------------------+
2. Core Problem & Failure Modes
Operating petabyte-scale storage engines introduces severe operational pitfalls:
-
The Cross-Shard Scatter-Gather Explosion in Vitess: If an application executes
SELECT * FROM orders WHERE order_status = 'PENDING'andorder_statusis not the sharding key, VTGate cannot route the query to a single shard. VTGate must scatter the query across all 256 MySQL shards in parallel, gather all responses, sort/deduplicate them in memory, and stream the result. A single scatter-gather query can saturate network bandwidth and freeze application response pools. -
The “Too Many Parts” Write Rejection in ClickHouse: ClickHouse’s MergeTree is an append-only engine where every
INSERTtransaction writes a new immutable physical directory (a “part”) to disk. Background threads merge parts asynchronously. If an application streams unbatched single-row inserts (e.g., 5,000 writes/sec), part creation completely outpaces merge throughput, causing ClickHouse to throwToo many partsand halt write ingestion. -
MySQL Connection Exhaustion Without VTTablet: MySQL’s thread-per-connection architecture scales poorly past a few hundred connections. If 500 application pods each open 20 connections, MySQL faces 10,000 concurrent threads, spending of CPU time on thread scheduling and mutex contention.
-
Sparse Index Grain Misses in ClickHouse: ClickHouse does not build traditional B-Tree row-level indexes. It uses a Sparse Primary Index with a default granularity of 8,192 rows. If a developer attempts to use ClickHouse as a key-value store looking up unique single records via non-primary keys, ClickHouse must scan tens of thousands of rows off disk, resulting in poor point-lookup performance.
Foundational Bridge: Distributed Batch Processing (Spark Shuffles) & Open Data Lakehouses (Parquet + Iceberg)
Complementing real-time OLTP sharding (Vitess) and sub-second OLAP engines (ClickHouse), petabyte-scale analytical pipelines rely on Distributed Batch Compute and Open Lakehouse Table Formats:
- MapReduce vs. Apache Spark DAG Execution (Narrow vs. Wide Transformations):
- Narrow Transformations (
map,filter,union): Each output partition depends on at most one input partition. Spark fuses narrow stages into a single pipelined in-memory pass with zero network I/O. - Wide Transformations (Shuffle Boundaries) (
groupByKey,reduceByKey,join): Output partitions depend on records scattered across all upstream partitions. Map tasks hash-partition records into shuffle spill files on local NVMe, and Reduce tasks fetch blocks across the network—making network shuffle bandwidth and skewed keys the primary bottleneck.
- Narrow Transformations (
- Columnar Storage Encoding (
Apache Parquet& Dremel Shredding):- Parquet horizontally partitions tables into Row Groups (), stores each column contiguously as Column Chunks split into Pages, and embeds min/max statistics and dictionary/RLE/Bit-Packing encodings in the footer. Query engines skip entire Row Groups via Predicate Pushdown without reading irrelevant bytes from object storage.
- Nested schemas (arrays/structs) are losslessly flattened into columnar stripes using Google Dremel’s Repetition Levels and Definition Levels.
- Open Lakehouse ACID Table Formats (
Apache Iceberg):- Raw Parquet directories on S3 lack atomic multi-file commits and suffer slow
LISToperations. - Apache Iceberg introduces a 3-tier metadata tree: Catalog Pointer Metadata File (table schema, partition spec, snapshot history) Manifest List (one per snapshot) Manifest Files (tracking individual Parquet data files and delete files with column-level bounds).
- Commits execute via atomic Compare-And-Swap on the Catalog root pointer, enabling Serializable Snapshot Isolation, Time Travel queries, Hidden Partitioning, and zero-copy schema evolution directly on cloud object storage.
- Raw Parquet directories on S3 lack atomic multi-file commits and suffer slow
3. Technical Deep Dive
Vitess Sharding Architecture & VSchema
Vitess sits between application servers and a fleet of MySQL instances. It presents a unified, single-database abstraction:
[Application] --(MySQL Protocol)--> [VTGate Proxy]
|
+-----------------+-----------------+
| (VSchema Hash: keyspace_id) |
v v
[VTTablet: Shard -80] [VTTablet: Shard 80-]
| |
[MySQL Primary] [MySQL Primary]
- VTGate: A lightweight, stateless proxy. It parses incoming SQL, consults the VSchema (Vitess Schema) to calculate the target shard, rewrites the query, routes it to the correct backend, and aggregates partial results.
- VSchema: Defines how tables are partitioned. A table specifies a Primary Vindex (e.g.,
user_idusing anxxhashormd5keyspace ID generator). The continuous keyspace ranges from0x00to0xFF.- Shard 1 covers range
[-80)(tokens from0x00to0x7F). - Shard 2 covers range
[80-)(tokens from0x80to0xFF).
- Shard 1 covers range
- VTTablet: A dedicated sidecar managing a single MySQL instance. It maintains a warm connection pool (converting 5,000 client connections into 50 MySQL connections), cancels queries exceeding execution deadlines, and consolidates identical concurrent reads.
ClickHouse MergeTree Storage Engine
ClickHouse stores data on disk in columnar format using the MergeTree family of engines.
Physical Column Layout on Disk
For a table with columns timestamp, user_id, latency:
- Each column is physically stored in separate compressed files:
timestamp.bin,user_id.bin,latency.bin. - Because identical data types sit contiguously on disk, ClickHouse achieves massive compression ratios () using LZ4, ZSTD, and specialized double-delta and T64 codecs.
Primary Key Sparse Index
Unlike MySQL, which indexes every single row in a B-Tree, ClickHouse indexes one row every rows (default: rows, called an index mark):
For a table with 1 billion rows, the primary index contains only entries, fitting entirely in a few megabytes of RAM! When a query filters by primary key, ClickHouse uses the sparse index to binary-search the matching mark blocks, reading only the necessary 8,192-row chunks off disk.
SIMD Vectorized Query Execution
ClickHouse processes data in vectors (arrays of up to 65,536 values in memory). When computing sum(latency), ClickHouse feeds contiguous 64-bit float arrays into modern CPU SIMD registers (AVX-512), computing 8 to 16 floating-point additions in a single CPU instruction clock cycle!
4. System Design Patterns & Trade-offs
+----------------------------------------------------------------------------------------------------+
| TRADEOFF MATRIX: LARGE-SCALE STORAGE PARADIGMS |
+--------------------------+------------------------------------+------------------------------------+
| Dimension | Vitess (Sharded MySQL) | ClickHouse (Columnar MergeTree) |
+--------------------------+------------------------------------+------------------------------------+
| Target Workload | Mission-critical OLTP transactions | Real-time Big Data OLAP analytics |
| Point Lookup by PK | Sub-millisecond (B-Tree on shard) | ~10-30ms (Scans 8192-row mark block|
| Aggregation across 100M | Very Slow (Cross-shard joins/scans)| Extremely Fast (500M rows/sec/core)|
| ACID Transactions | Supported (Per-shard ACID, 2PC) | No multi-statement transactions |
| Ingestion Pattern | High-frequency single-row updates | Batched appends (10k+ rows/insert) |
| Secondary Indexes | Supported (Lookup Vindexes) | MinMax skip indexes, projection ind|
+--------------------------+------------------------------------+------------------------------------+
5. Practical Implementation
Below is a complete TypeScript simulation demonstrating:
- Vitess VSchema query router that calculates hexadecimal keyspace IDs and routes queries to targeted shards.
- ClickHouse Columnar Storage Engine with physical column slicing, sparse mark indexing, and vectorized array aggregation.
import { createHash } from 'crypto';
/**
* -------------------------------------------------------------
* 1. VITESS SHARDING SIMULATOR
* -------------------------------------------------------------
*/
export interface ShardRange {
name: string;
startHex: string;
endHex: string;
}
export class VitessRouter {
private shards: ShardRange[] = [
{ name: 'shard_-80', startHex: '00', endHex: '80' },
{ name: 'shard_80-', startHex: '80', endHex: 'FF' }
];
public hashKeyToKeyspaceId(routingKey: string): string {
return createHash('md5').update(routingKey).digest('hex').substring(0, 2).toUpperCase();
}
public routeQuery(routingKey: string): string {
const keyspaceId = this.hashKeyToKeyspaceId(routingKey);
const target = this.shards.find((s) => keyspaceId >= s.startHex && keyspaceId < s.endHex);
return target ? target.name : this.shards[this.shards.length - 1].name;
}
}
/**
* -------------------------------------------------------------
* 2. CLICKHOUSE COLUMNAR MERGETREE ENGINE SIMULATOR
* -------------------------------------------------------------
*/
export class ClickHouseColumnarTable {
// Physical column separation on disk/memory
private colUserId: number[] = [];
private colLatencyMs: number[] = [];
private colStatusCode: number[] = [];
// Sparse index marks (one entry every 8192 rows, simulated as 4 rows for demo)
private readonly indexGranularity = 4;
private sparseIndex: { markRow: number; userId: number }[] = [];
public insertBatch(records: { userId: number; latencyMs: number; statusCode: number }[]): void {
// Records must be sorted by primary key (userId) before insertion
records.sort((a, b) => a.userId - b.userId);
for (const record of records) {
const currentRow = this.colUserId.length;
if (currentRow % this.indexGranularity === 0) {
this.sparseIndex.push({ markRow: currentRow, userId: record.userId });
}
this.colUserId.push(record.userId);
this.colLatencyMs.push(record.latencyMs);
this.colStatusCode.push(record.statusCode);
}
}
/**
* Vectorized Aggregation: SELECT avg(latencyMs) WHERE statusCode = 500
* Reads ONLY the latency and status columns; ignores userId!
*/
public executeVectorizedAvgLatency(filterStatusCode: number): number {
let sum = 0;
let count = 0;
const n = this.colStatusCode.length;
// Sequential loop over contiguous memory simulates SIMD processing
for (let i = 0; i < n; i++) {
if (this.colStatusCode[i] === filterStatusCode) {
sum += this.colLatencyMs[i];
count++;
}
}
return count > 0 ? sum / count : 0;
}
public getSparseIndexSize(): number {
return this.sparseIndex.length;
}
}
function runDemo(): void {
console.log("=== Vitess & ClickHouse Storage Engine Architecture ===");
const router = new VitessRouter();
console.log(
`Vitess hash routing: 'usr_42' (ksid=0x${router.hashKeyToKeyspaceId("usr_42")}) -> ${router.routeQuery("usr_42")}, ` +
`'usr_99' (ksid=0x${router.hashKeyToKeyspaceId("usr_99")}) -> ${router.routeQuery("usr_99")}`
);
const clickhouse = new ClickHouseColumnarTable();
clickhouse.insertBatch([
{ userId: 101, latencyMs: 23, statusCode: 200 },
{ userId: 102, latencyMs: 154, statusCode: 500 },
{ userId: 103, latencyMs: 206, statusCode: 500 },
{ userId: 104, latencyMs: 18, statusCode: 200 },
{ userId: 105, latencyMs: 31, statusCode: 200 },
]);
console.log(
`ClickHouse MergeTree: inserted 5 rows (sparseIndexMarks=${clickhouse.getSparseIndexSize()}), ` +
`vectorized avg(latencyMs) for status=500 -> ${clickhouse.executeVectorizedAvgLatency(500)}ms`
);
}
runDemo();
Simulation Execution Output & State Transitions
=== Vitess & ClickHouse Storage Engine Architecture ===
Vitess hash routing: 'usr_42' (ksid=0x5C) -> shard_-80, 'usr_99' (ksid=0xA9) -> shard_80-
ClickHouse MergeTree: inserted 5 rows (sparseIndexMarks=2), vectorized avg(latencyMs) for status=500 -> 180ms
6. Failure Scenarios & Edge Cases
1. Scatter-Gather Query Thundering Herd in Vitess
A developer deploys a new query: SELECT * FROM payments WHERE created_at >= NOW() - INTERVAL 1 HOUR;. Because created_at is not a sharding key, VTGate must dispatch 512 simultaneous parallel queries to all 512 shards. When 500 web requests execute this query concurrently, 256,000 MySQL queries launch at once, saturating database connections and CPU.
- Mitigation: Use
Lookup Vindexesto index non-primary keys, or replicate historical data into ClickHouse for range-based reporting.
2. ClickHouse Memory Exhaustion on Unbounded High-Cardinality GROUP BY
An analyst runs SELECT user_ip, user_agent, count(*) FROM clickstream GROUP BY user_ip, user_agent; across 5 billion rows. ClickHouse must instantiate a hash table in RAM to track every unique (user_ip, user_agent) combination. When the cardinality exceeds 100 million distinct keys, the hash table exceeds server RAM, triggering an instant memory limit exception (Memory limit (for query) exceeded).
- Mitigation: Set
max_bytes_before_external_group_byto spill large hash tables to disk, or use approximate aggregation algorithms likeuniqCombined()/HyperLogLog.
7. Production Operations & Observability
Vitess VReplication Cutover Protocol
To execute a zero-downtime resharding cutover:
- Validate replication lag:
vtctlclient VReplicationExec ShardTarget "SHOW VREPLICATION"(confirm lag ). - Execute traffic switch:
vtctlclient SwitchTraffic -tablet_type=primary KeyspaceName.WorkflowName. - VTGate pauses queries for , commits the metadata switch, and resumes writes on the new target shards.
8. Security & Compliance
- Vitess Table-Level ACLs: Configure VTTablet authorization files to restrict access to sensitive payment and authentication tables strictly to designated microservice service accounts.
- ClickHouse Row-Level Security (RLS): Enforce tenant isolation in multi-tenant analytics using ClickHouse RLS policies:
CREATE ROW POLICY tenant_policy ON analytics_table FOR SELECT USING (tenant_id = current_user()).
9. Real-World Case Studies
YouTube Scaling MySQL with Vitess
YouTube originally powered video metadata and user interactions with a massive monolithic MySQL master.
- When write load exceeded single-server limits, YouTube engineers created Vitess to shard MySQL horizontally.
- Today, Vitess powers hundreds of thousands of MySQL shards serving millions of QPS, enabling YouTube to perform online resharding with zero application downtime.
Cloudflare Analytics on ClickHouse
Cloudflare logs DNS requests, HTTP traffic, and DDoS mitigation events globally.
- They ingest over into ClickHouse clusters.
- Because ClickHouse compresses data by and evaluates aggregations across billions of rows in milliseconds, Cloudflare delivers real-time security dashboards to customers at fraction of the cost of Elasticsearch.
10. Technology Comparison Matrix
+----------------------------------------------------------------------------------------------------+
| MASSIVE-SCALE STORAGE PLATFORMS |
+----------------------+--------------------+--------------------+-----------------------------------+
| Platform | Data Model | Query Engine | Primary Superpower |
+----------------------+--------------------+--------------------+-----------------------------------+
| Vitess | Relational (Row) | Sharded MySQL Proxy| Zero-downtime sharding of OLTP SQL|
+----------------------+--------------------+--------------------+-----------------------------------+
| ClickHouse | Columnar (OLAP) | Vectorized SIMD | Blazing fast real-time aggregates |
+----------------------+--------------------+--------------------+-----------------------------------+
| Citus (PostgreSQL) | Relational (Row) | Sharded Postgres | Native PostgreSQL extensions |
+----------------------+--------------------+--------------------+-----------------------------------+
| Apache Druid | Columnar + Bitmap | Distributed OLAP | Streaming real-time ingestion |
+----------------------+--------------------+--------------------+-----------------------------------+
| Snowflake | Columnar Cloud DWH | Decoupled S3 Engine| Ad-hoc enterprise SQL warehousing |
+----------------------+--------------------+--------------------+-----------------------------------+
11. Design Challenge & Exercise
Scenario: High-Volume Ride-Hailing Analytics Pipeline
You are designing the data platform for a global ride-hailing company processing across 50 cities.
Requirements:
- Active ride dispatching and payment updates require strict ACID transactions on MySQL.
- Live driver tracking analytics require computing real-time surge pricing and demand heatmaps across 500,000,000 location pings every 5 minutes.
Your Architecture Deliverables:
- Outline the Vitess VSchema sharding strategy for the core
ridesandridersOLTP tables. - Design the CDC (Change Data Capture) pipeline streaming completed rides from Vitess into ClickHouse.
- Draft the ClickHouse
ENGINE = MergeTree()table schema and partition key for driver location telemetry.
View Reference Architecture & Worked Solutions (CORE-M36-001)
1. Vitess VSchema Sharding Strategy for OLTP riders and rides Tables
- Keyspace & Primary Vindex Selection:
- At (plus during active trips), provision a sharded Vitess keyspace
ride_coresplit into shards (-04,04-08, ,fc-). - Shard both
ridersandridesbyrider_idusing thexxhash(orbinary_md5) Primary Vindex so a rider’s profile, active trip, and payment history are co-located on the exact same MySQL shard, enabling single-shard ACID transactions without two-phase commit: - Because drivers and trip-lookup APIs frequently query
SELECT * FROM rides WHERE ride_id = ?orWHERE driver_id = ? AND status = 'ACTIVE', define a Consistent Lookup Vindex (ride_id_lookupanddriver_active_ride_lookup) soVTGateresolvesride_id -> keyspace_idin without a 64-shard scatter-gather:
- At (plus during active trips), provision a sharded Vitess keyspace
{
"sharded": true,
"vindexes": {
"xxhash_vdx": { "type": "xxhash" },
"ride_id_lookup": {
"type": "consistent_lookup_unique",
"params": {
"table": "ride_core.ride_id_map",
"from": "ride_id",
"to": "keyspace_id"
},
"owner": "rides"
}
},
"tables": {
"riders": {
"column_vindexes": [{ "column": "rider_id", "name": "xxhash_vdx" }]
},
"rides": {
"column_vindexes": [
{ "column": "rider_id", "name": "xxhash_vdx" },
{ "column": "ride_id", "name": "ride_id_lookup" }
]
}
}
}
2. Zero-Loss CDC Pipeline from Vitess to ClickHouse (Avoiding Too many parts)
- Stream Extraction via Vitess
VStream:- Rather than connecting 64 separate binlog readers directly to underlying MySQL instances (which breaks during Vitess
ReshardorMoveTablescutovers), consume the unified VitessVStreamgRPC API via the Debezium Vitess Connector onVTGate.VStreamtransparently tracks shard splits and emits GTID-ordered row change events into an Apache Kafka topicride_events_cdc(partitioned bycity_id).
- Rather than connecting 64 separate binlog readers directly to underlying MySQL instances (which breaks during Vitess
- Micro-Batched ClickHouse Ingestion:
- Writing directly to ClickHouse triggers
DB::Exception: Too many parts. Instead, use ClickHouse’s nativeKafkatable engine paired with a Materialized View (or the ClickHouse Kafka Connect Sink) configured withstream_flush_interval_ms = 2000andmax_insert_block_size = 1048576: - Every , each ClickHouse shard flushes a single sorted bulk block of rows into a
ReplacingMergeTree(updated_at_ms)table, keeping part creation at while deduplicating mutable ride state transitions (REQUESTEDIN_PROGRESSCOMPLETED).
- Writing directly to ClickHouse triggers
3. ClickHouse MergeTree Schema & Pre-Aggregated Materialized View for Driver Pings / 5 Min
- Throughput Envelope: .
- Physical Table & Sparse Primary Key (
ORDER BY) Design:- Order the
MergeTreetable from lowest cardinality to highest cardinality matching the surge-pricing query predicates (city_id, geospatialh3_index_r8, 5-minute window,driver_id). Partition by day (toYYYYMMDD(ping_ts)) so each partition holds rows across the cluster (e.g., ClickHouse shards) and old partitions drop cleanly viaDROP PARTITION:
- Order the
CREATE TABLE driver_pings_local ON CLUSTER 'analytics_cluster' (
ping_ts DateTime64(3, 'UTC') CODEC(DoubleDelta, ZSTD(1)),
city_id UInt16 CODEC(T64, LZ4),
h3_cell_res8 UInt64 CODEC(T64, LZ4),
driver_id UInt64 CODEC(ZSTD(1)),
lat Float32 CODEC(Gorilla, LZ4),
lon Float32 CODEC(Gorilla, LZ4),
is_available UInt8 CODEC(T64, LZ4),
speed_kph Float32 CODEC(Gorilla, LZ4)
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/driver_pings', '{replica}')
PARTITION BY toYYYYMMDD(ping_ts)
ORDER BY (city_id, h3_cell_res8, toStartOfFiveMinutes(ping_ts), driver_id)
SETTINGS index_granularity = 8192;
-- Real-Time Surge Pricing Rollup via Incremental Materialized View:
CREATE MATERIALIZED VIEW driver_supply_5m_mv ON CLUSTER 'analytics_cluster'
ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/driver_supply_5m', '{replica}')
PARTITION BY toYYYYMM(window_start)
ORDER BY (city_id, h3_cell_res8, window_start)
AS SELECT
city_id,
h3_cell_res8,
toStartOfFiveMinutes(ping_ts) AS window_start,
uniqCombined64State(driver_id) AS unique_drivers_state,
uniqCombined64IfState(driver_id, is_available = 1) AS available_drivers_state,
avgState(speed_kph) AS avg_speed_state
FROM driver_pings_local
GROUP BY city_id, h3_cell_res8, window_start;
- Why This Schema Meets Sub-Second SLA: Rather than re-scanning raw rows on every dashboard refresh,
driver_supply_5m_mvincrementally updates HyperLogLog (uniqCombined64State) andavgStatesketches in memory as each batch is inserted, allowing surge-pricing services to query all H3 hex cells in a city in .
12. Assessment
Review the 4 scenario-based quiz questions at the top of this lesson to test your mastery of Vitess MySQL sharding and ClickHouse columnar vectorized analytics.
13. Summary & Architectural Decision Guide: Workload Family 6 Deep Dive
Analytical (OLAP), Search & Graph Workload Architecture
- Columnar Layouts & Compression:
- Store each column in separate files compressed with LZ4 or ZSTD.
- Columnar layout enables 5x–10x compression ratios and prunes 90%+ of disk I/O on aggregations.
- Sparse Primary Index Granularity:
- Sample 1 index mark per 8,192 rows (
index_granularity). The entire index sits in RAM. - Query filters quickly skip non-matching granules without random disk head seeks.
- Sample 1 index mark per 8,192 rows (
- SIMD Vectorized Scans:
- Process arrays of primitives with AVX-512/NEON instructions in L1/L2 cache, executing billions of row operations per second per CPU core.
- Distributed Join & Resource Isolation:
- Avoid large distributed cross-node shuffles; prefer pre-aggregated materialized views or denormalized wide tables.
- Enforce CPU and memory query quotas (
max_memory_usage) to isolate ad-hoc analyst queries from operational real-time dashboards.
14. Authoritative References & Version Context
- Alexey Milovidov et al. (2016), “ClickHouse: Fast Open-Source Column-Oriented Database Management System”, Yandex / ClickHouse Technical Architecture — clickhouse.com/docs/en/development/architecture
- Peter Zaitsev and Vadim Tkachenko (2020), “High Performance MySQL: Optimization, Backups, and Replication”, O’Reilly Media — oreilly.com
- Daniel Abadi, Peter Boncz, and Stavros Harizopoulos (2009), “Column-Oriented Database Systems”, VLDB ‘09 / Foundations and Trends in Databases — doi.org/10.1561/1900000024
- Version Context: Architecture reflects ClickHouse v24.3 LTS (MergeTree engine, SIMD vectorization) and Vitess v19.0 LTS (VReplication, VTGate distributed query planner).