By the end of this lesson
  • Deconstruct Vitess's transparent horizontal sharding: VTGate intelligent query routing, VTTablet connection management, and VSchema keyspace definitions
  • Master VReplication workflows: Zero-downtime resharding cutovers, online schema migrations, and materialization
  • Analyze ClickHouse's vectorized columnar storage: MergeTree engine families, primary key sparse indexes, and SIMD instruction parallelism
  • Implement an algorithmic TypeScript Vitess VSchema router and ClickHouse columnar aggregation engine

1. Context & Architecture Reality

Project Identity, Version & License Baseline

Project ComponentCanonical RepositoryPinned ReleaseGovernance / LicenseEdition Boundary
Vitessvitessio/vitessv19.0 (LTS)CNCF Graduated (Apache 2.0)Open source
ClickHouseClickHouse/ClickHousev24.3 (LTS)ClickHouse Inc (Apache 2.0)Open source
SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED SYSTEMS
Large-Scale Storage Architecture: Vitess Sharding and ClickHouse OLAP
ARCHITECTURAL FLOW · STEP 1 OF 3

1. Shard 1: Keyspace Range [-80) & Shard 2: Keyspace Range [80-)

Establishes the ingress and structural boundary across Shard 1: Keyspace Range [-80) & Shard 2: Keyspace Range [80-): VTTablet → MySQL Primary + Replicas Connection pooling, query killer, row cache; Streams binlog mutations for zero-downtime resharding and split cutovers in < 1s!; Index granularity = 8192 rows 1 index entry per mark block Entire index fits in RAM! Zero random disk seeks.

Architectural Invariant: AVX-512 CPU registers process 16 columns in a single instruction Aggregate 500M rows / sec! Pure hardware-accelerated math.
ARCHITECTURAL FLOW · STEP 2 OF 3

2. Row Store (MySQL / Postgres) & Column Store (ClickHouse)

Coordinates core data-plane execution and state transitions through Row Store (MySQL / Postgres) & Column Store (ClickHouse): [Row1: id, name, age, bio, addr]; [Row2: id, name, age, bio, addr]; Col Age: [25, 34, 19, 42, 55...].

Architectural Invariant: Latency: Sub-10ms p99 transactions.
ARCHITECTURAL FLOW · STEP 3 OF 3

3. ClickHouse: Vectorized Columnar MergeTree & Physical Data Layout: Row Store vs Column Store

Enforces durability, state convergence, and output verification across ClickHouse: Vectorized Columnar MergeTree, Physical Data Layout: Row Store vs Column Store, OLTP vs OLAP Architectural Tradeoffs within the Large-Scale Storage Architecture: Vitess Sharding and ClickHouse OLAP architecture.

Architectural Invariant: Maintains deterministic distributed systems invariants across ClickHouse: Vectorized Columnar MergeTree & Physical Data Layout: Row Store vs Column Store.

At moderate scale (tens of gigabytes, a few thousand queries per second), a single well-tuned relational database server running PostgreSQL or MySQL handles application traffic easily. However, when an enterprise scales to tens of terabytes, hundreds of millions of users, or billions of events per day, single-node architectures reach a hard physical limit.

At hyper-scale, data workloads bifurcate into two diametrically opposed access patterns:

  1. Online Transaction Processing (OLTP): High-concurrency, low-latency, point reads and writes on individual user entities (e.g., user profiles, shopping carts, checkout payments). Requirements: ACID transactions, row-level locking, sub-10ms response times.
  2. Online Analytical Processing (OLAP): High-throughput aggregations, trend analysis, and reporting across billions of events (e.g., telemetry, log analytics, security audit trails, BI dashboards). Requirements: Scanning billions of rows per second, complex GROUP BY and AVG operations.

No single database engine can optimize both patterns simultaneously. Attempting to run real-time analytical queries on a transactional MySQL primary locks tables, exhausts memory buffers, and tanks user checkout transactions.

This architecture study deconstructs the two premier open-source solutions designed for each extreme:

  • Vitess: The horizontal scaling engine created by YouTube to shard and manage thousands of MySQL nodes transparently.
  • ClickHouse: The ultra-fast columnar analytical engine designed to aggregate hundreds of millions of rows per second per server.
+--------------------------------------------------------------------------------------------------+
|                            THE TWO POLES OF LARGE-SCALE STORAGE                                  |
|                                                                                                  |
|   OLTP TRANSACTIONAL WORKLOAD (Vitess)             OLAP ANALYTICAL WORKLOAD (ClickHouse)         |
|                                                    |                                             |
|   - Workload: Single-row INSERT / UPDATE           | - Workload: Batched bulk appends, wide scans|
|   - Concurrency: 100,000 QPS                       | - Concurrency: 50 - 500 QPS                 |
|   - Query: SELECT * FROM users WHERE id = 4821;    | - Query: SELECT avg(latency) FROM logs      |
|   - Latency SLA: < 10ms                            |   WHERE status = 500 GROUP BY country;      |
|   - Storage Engine: Row-Store (MySQL InnoDB)       | - Latency SLA: 200ms - 2s (across 10B rows!)|
|   - Architecture: Transparent Sharding via VTGate  | - Storage Engine: Columnar (MergeTree)      |
+--------------------------------------------------------------------------------------------------+

2. Core Problem & Failure Modes

Operating petabyte-scale storage engines introduces severe operational pitfalls:

  1. The Cross-Shard Scatter-Gather Explosion in Vitess: If an application executes SELECT * FROM orders WHERE order_status = 'PENDING' and order_status is not the sharding key, VTGate cannot route the query to a single shard. VTGate must scatter the query across all 256 MySQL shards in parallel, gather all responses, sort/deduplicate them in memory, and stream the result. A single scatter-gather query can saturate network bandwidth and freeze application response pools.

  2. The “Too Many Parts” Write Rejection in ClickHouse: ClickHouse’s MergeTree is an append-only engine where every INSERT transaction writes a new immutable physical directory (a “part”) to disk. Background threads merge parts asynchronously. If an application streams unbatched single-row inserts (e.g., 5,000 writes/sec), part creation completely outpaces merge throughput, causing ClickHouse to throw Too many parts and halt write ingestion.

  3. MySQL Connection Exhaustion Without VTTablet: MySQL’s thread-per-connection architecture scales poorly past a few hundred connections. If 500 application pods each open 20 connections, MySQL faces 10,000 concurrent threads, spending of CPU time on thread scheduling and mutex contention.

  4. Sparse Index Grain Misses in ClickHouse: ClickHouse does not build traditional B-Tree row-level indexes. It uses a Sparse Primary Index with a default granularity of 8,192 rows. If a developer attempts to use ClickHouse as a key-value store looking up unique single records via non-primary keys, ClickHouse must scan tens of thousands of rows off disk, resulting in poor point-lookup performance.


Foundational Bridge: Distributed Batch Processing (Spark Shuffles) & Open Data Lakehouses (Parquet + Iceberg)

Complementing real-time OLTP sharding (Vitess) and sub-second OLAP engines (ClickHouse), petabyte-scale analytical pipelines rely on Distributed Batch Compute and Open Lakehouse Table Formats:

  1. MapReduce vs. Apache Spark DAG Execution (Narrow vs. Wide Transformations):
    • Narrow Transformations (map, filter, union): Each output partition depends on at most one input partition. Spark fuses narrow stages into a single pipelined in-memory pass with zero network I/O.
    • Wide Transformations (Shuffle Boundaries) (groupByKey, reduceByKey, join): Output partitions depend on records scattered across all upstream partitions. Map tasks hash-partition records into shuffle spill files on local NVMe, and Reduce tasks fetch blocks across the network—making network shuffle bandwidth and skewed keys the primary bottleneck.
  2. Columnar Storage Encoding (Apache Parquet & Dremel Shredding):
    • Parquet horizontally partitions tables into Row Groups (), stores each column contiguously as Column Chunks split into Pages, and embeds min/max statistics and dictionary/RLE/Bit-Packing encodings in the footer. Query engines skip entire Row Groups via Predicate Pushdown without reading irrelevant bytes from object storage.
    • Nested schemas (arrays/structs) are losslessly flattened into columnar stripes using Google Dremel’s Repetition Levels and Definition Levels.
  3. Open Lakehouse ACID Table Formats (Apache Iceberg):
    • Raw Parquet directories on S3 lack atomic multi-file commits and suffer slow LIST operations.
    • Apache Iceberg introduces a 3-tier metadata tree: Catalog Pointer Metadata File (table schema, partition spec, snapshot history) Manifest List (one per snapshot) Manifest Files (tracking individual Parquet data files and delete files with column-level bounds).
    • Commits execute via atomic Compare-And-Swap on the Catalog root pointer, enabling Serializable Snapshot Isolation, Time Travel queries, Hidden Partitioning, and zero-copy schema evolution directly on cloud object storage.

3. Technical Deep Dive

Vitess Sharding Architecture & VSchema

Vitess sits between application servers and a fleet of MySQL instances. It presents a unified, single-database abstraction:

[Application] --(MySQL Protocol)--> [VTGate Proxy]
                                          |
                        +-----------------+-----------------+
                        | (VSchema Hash: keyspace_id)       |
                        v                                   v
             [VTTablet: Shard -80]                [VTTablet: Shard 80-]
                        |                                   |
                [MySQL Primary]                     [MySQL Primary]
  1. VTGate: A lightweight, stateless proxy. It parses incoming SQL, consults the VSchema (Vitess Schema) to calculate the target shard, rewrites the query, routes it to the correct backend, and aggregates partial results.
  2. VSchema: Defines how tables are partitioned. A table specifies a Primary Vindex (e.g., user_id using an xxhash or md5 keyspace ID generator). The continuous keyspace ranges from 0x00 to 0xFF.
    • Shard 1 covers range [-80) (tokens from 0x00 to 0x7F).
    • Shard 2 covers range [80-) (tokens from 0x80 to 0xFF).
  3. VTTablet: A dedicated sidecar managing a single MySQL instance. It maintains a warm connection pool (converting 5,000 client connections into 50 MySQL connections), cancels queries exceeding execution deadlines, and consolidates identical concurrent reads.

ClickHouse MergeTree Storage Engine

ClickHouse stores data on disk in columnar format using the MergeTree family of engines.

Physical Column Layout on Disk

For a table with columns timestamp, user_id, latency:

  • Each column is physically stored in separate compressed files: timestamp.bin, user_id.bin, latency.bin.
  • Because identical data types sit contiguously on disk, ClickHouse achieves massive compression ratios () using LZ4, ZSTD, and specialized double-delta and T64 codecs.

Primary Key Sparse Index

Unlike MySQL, which indexes every single row in a B-Tree, ClickHouse indexes one row every rows (default: rows, called an index mark):

For a table with 1 billion rows, the primary index contains only entries, fitting entirely in a few megabytes of RAM! When a query filters by primary key, ClickHouse uses the sparse index to binary-search the matching mark blocks, reading only the necessary 8,192-row chunks off disk.

SIMD Vectorized Query Execution

ClickHouse processes data in vectors (arrays of up to 65,536 values in memory). When computing sum(latency), ClickHouse feeds contiguous 64-bit float arrays into modern CPU SIMD registers (AVX-512), computing 8 to 16 floating-point additions in a single CPU instruction clock cycle!


4. System Design Patterns & Trade-offs

+----------------------------------------------------------------------------------------------------+
| TRADEOFF MATRIX: LARGE-SCALE STORAGE PARADIGMS                                                     |
+--------------------------+------------------------------------+------------------------------------+
| Dimension                | Vitess (Sharded MySQL)             | ClickHouse (Columnar MergeTree)    |
+--------------------------+------------------------------------+------------------------------------+
| Target Workload          | Mission-critical OLTP transactions | Real-time Big Data OLAP analytics  |
| Point Lookup by PK       | Sub-millisecond (B-Tree on shard)  | ~10-30ms (Scans 8192-row mark block|
| Aggregation across 100M  | Very Slow (Cross-shard joins/scans)| Extremely Fast (500M rows/sec/core)|
| ACID Transactions        | Supported (Per-shard ACID, 2PC)    | No multi-statement transactions    |
| Ingestion Pattern        | High-frequency single-row updates  | Batched appends (10k+ rows/insert) |
| Secondary Indexes        | Supported (Lookup Vindexes)        | MinMax skip indexes, projection ind|
+--------------------------+------------------------------------+------------------------------------+

5. Practical Implementation

Below is a complete TypeScript simulation demonstrating:

  1. Vitess VSchema query router that calculates hexadecimal keyspace IDs and routes queries to targeted shards.
  2. ClickHouse Columnar Storage Engine with physical column slicing, sparse mark indexing, and vectorized array aggregation.
import { createHash } from 'crypto';

/**
 * -------------------------------------------------------------
 * 1. VITESS SHARDING SIMULATOR
 * -------------------------------------------------------------
 */
export interface ShardRange {
  name: string;
  startHex: string;
  endHex: string;
}

export class VitessRouter {
  private shards: ShardRange[] = [
    { name: 'shard_-80', startHex: '00', endHex: '80' },
    { name: 'shard_80-', startHex: '80', endHex: 'FF' }
  ];

  public hashKeyToKeyspaceId(routingKey: string): string {
    return createHash('md5').update(routingKey).digest('hex').substring(0, 2).toUpperCase();
  }

  public routeQuery(routingKey: string): string {
    const keyspaceId = this.hashKeyToKeyspaceId(routingKey);
    const target = this.shards.find((s) => keyspaceId >= s.startHex && keyspaceId < s.endHex);
    return target ? target.name : this.shards[this.shards.length - 1].name;
  }
}

/**
 * -------------------------------------------------------------
 * 2. CLICKHOUSE COLUMNAR MERGETREE ENGINE SIMULATOR
 * -------------------------------------------------------------
 */
export class ClickHouseColumnarTable {
  // Physical column separation on disk/memory
  private colUserId: number[] = [];
  private colLatencyMs: number[] = [];
  private colStatusCode: number[] = [];

  // Sparse index marks (one entry every 8192 rows, simulated as 4 rows for demo)
  private readonly indexGranularity = 4;
  private sparseIndex: { markRow: number; userId: number }[] = [];

  public insertBatch(records: { userId: number; latencyMs: number; statusCode: number }[]): void {
    // Records must be sorted by primary key (userId) before insertion
    records.sort((a, b) => a.userId - b.userId);

    for (const record of records) {
      const currentRow = this.colUserId.length;
      if (currentRow % this.indexGranularity === 0) {
        this.sparseIndex.push({ markRow: currentRow, userId: record.userId });
      }

      this.colUserId.push(record.userId);
      this.colLatencyMs.push(record.latencyMs);
      this.colStatusCode.push(record.statusCode);
    }
  }

  /**
   * Vectorized Aggregation: SELECT avg(latencyMs) WHERE statusCode = 500
   * Reads ONLY the latency and status columns; ignores userId!
   */
  public executeVectorizedAvgLatency(filterStatusCode: number): number {
    let sum = 0;
    let count = 0;

    const n = this.colStatusCode.length;
    // Sequential loop over contiguous memory simulates SIMD processing
    for (let i = 0; i < n; i++) {
      if (this.colStatusCode[i] === filterStatusCode) {
        sum += this.colLatencyMs[i];
        count++;
      }
    }

    return count > 0 ? sum / count : 0;
  }

  public getSparseIndexSize(): number {
    return this.sparseIndex.length;
  }
}

function runDemo(): void {
  console.log("=== Vitess & ClickHouse Storage Engine Architecture ===");
  const router = new VitessRouter();
  console.log(
    `Vitess hash routing: 'usr_42' (ksid=0x${router.hashKeyToKeyspaceId("usr_42")}) -> ${router.routeQuery("usr_42")}, ` +
    `'usr_99' (ksid=0x${router.hashKeyToKeyspaceId("usr_99")}) -> ${router.routeQuery("usr_99")}`
  );
  const clickhouse = new ClickHouseColumnarTable();
  clickhouse.insertBatch([
    { userId: 101, latencyMs: 23, statusCode: 200 },
    { userId: 102, latencyMs: 154, statusCode: 500 },
    { userId: 103, latencyMs: 206, statusCode: 500 },
    { userId: 104, latencyMs: 18, statusCode: 200 },
    { userId: 105, latencyMs: 31, statusCode: 200 },
  ]);
  console.log(
    `ClickHouse MergeTree: inserted 5 rows (sparseIndexMarks=${clickhouse.getSparseIndexSize()}), ` +
    `vectorized avg(latencyMs) for status=500 -> ${clickhouse.executeVectorizedAvgLatency(500)}ms`
  );
}

runDemo();

Simulation Execution Output & State Transitions

=== Vitess & ClickHouse Storage Engine Architecture ===
Vitess hash routing: 'usr_42' (ksid=0x5C) -> shard_-80, 'usr_99' (ksid=0xA9) -> shard_80-
ClickHouse MergeTree: inserted 5 rows (sparseIndexMarks=2), vectorized avg(latencyMs) for status=500 -> 180ms

6. Failure Scenarios & Edge Cases

1. Scatter-Gather Query Thundering Herd in Vitess

A developer deploys a new query: SELECT * FROM payments WHERE created_at >= NOW() - INTERVAL 1 HOUR;. Because created_at is not a sharding key, VTGate must dispatch 512 simultaneous parallel queries to all 512 shards. When 500 web requests execute this query concurrently, 256,000 MySQL queries launch at once, saturating database connections and CPU.

  • Mitigation: Use Lookup Vindexes to index non-primary keys, or replicate historical data into ClickHouse for range-based reporting.

2. ClickHouse Memory Exhaustion on Unbounded High-Cardinality GROUP BY

An analyst runs SELECT user_ip, user_agent, count(*) FROM clickstream GROUP BY user_ip, user_agent; across 5 billion rows. ClickHouse must instantiate a hash table in RAM to track every unique (user_ip, user_agent) combination. When the cardinality exceeds 100 million distinct keys, the hash table exceeds server RAM, triggering an instant memory limit exception (Memory limit (for query) exceeded).

  • Mitigation: Set max_bytes_before_external_group_by to spill large hash tables to disk, or use approximate aggregation algorithms like uniqCombined() / HyperLogLog.

7. Production Operations & Observability

Vitess VReplication Cutover Protocol

To execute a zero-downtime resharding cutover:

  1. Validate replication lag: vtctlclient VReplicationExec ShardTarget "SHOW VREPLICATION" (confirm lag ).
  2. Execute traffic switch: vtctlclient SwitchTraffic -tablet_type=primary KeyspaceName.WorkflowName.
  3. VTGate pauses queries for , commits the metadata switch, and resumes writes on the new target shards.

8. Security & Compliance

  1. Vitess Table-Level ACLs: Configure VTTablet authorization files to restrict access to sensitive payment and authentication tables strictly to designated microservice service accounts.
  2. ClickHouse Row-Level Security (RLS): Enforce tenant isolation in multi-tenant analytics using ClickHouse RLS policies: CREATE ROW POLICY tenant_policy ON analytics_table FOR SELECT USING (tenant_id = current_user()).

9. Real-World Case Studies

YouTube Scaling MySQL with Vitess

YouTube originally powered video metadata and user interactions with a massive monolithic MySQL master.

  • When write load exceeded single-server limits, YouTube engineers created Vitess to shard MySQL horizontally.
  • Today, Vitess powers hundreds of thousands of MySQL shards serving millions of QPS, enabling YouTube to perform online resharding with zero application downtime.

Cloudflare Analytics on ClickHouse

Cloudflare logs DNS requests, HTTP traffic, and DDoS mitigation events globally.

  • They ingest over into ClickHouse clusters.
  • Because ClickHouse compresses data by and evaluates aggregations across billions of rows in milliseconds, Cloudflare delivers real-time security dashboards to customers at fraction of the cost of Elasticsearch.

10. Technology Comparison Matrix

+----------------------------------------------------------------------------------------------------+
| MASSIVE-SCALE STORAGE PLATFORMS                                                                    |
+----------------------+--------------------+--------------------+-----------------------------------+
| Platform             | Data Model         | Query Engine       | Primary Superpower                |
+----------------------+--------------------+--------------------+-----------------------------------+
| Vitess               | Relational (Row)   | Sharded MySQL Proxy| Zero-downtime sharding of OLTP SQL|
+----------------------+--------------------+--------------------+-----------------------------------+
| ClickHouse           | Columnar (OLAP)    | Vectorized SIMD    | Blazing fast real-time aggregates |
+----------------------+--------------------+--------------------+-----------------------------------+
| Citus (PostgreSQL)   | Relational (Row)   | Sharded Postgres   | Native PostgreSQL extensions      |
+----------------------+--------------------+--------------------+-----------------------------------+
| Apache Druid         | Columnar + Bitmap  | Distributed OLAP   | Streaming real-time ingestion     |
+----------------------+--------------------+--------------------+-----------------------------------+
| Snowflake            | Columnar Cloud DWH | Decoupled S3 Engine| Ad-hoc enterprise SQL warehousing |
+----------------------+--------------------+--------------------+-----------------------------------+

11. Design Challenge & Exercise

Scenario: High-Volume Ride-Hailing Analytics Pipeline

You are designing the data platform for a global ride-hailing company processing across 50 cities.

Requirements:

  1. Active ride dispatching and payment updates require strict ACID transactions on MySQL.
  2. Live driver tracking analytics require computing real-time surge pricing and demand heatmaps across 500,000,000 location pings every 5 minutes.

Your Architecture Deliverables:

  • Outline the Vitess VSchema sharding strategy for the core rides and riders OLTP tables.
  • Design the CDC (Change Data Capture) pipeline streaming completed rides from Vitess into ClickHouse.
  • Draft the ClickHouse ENGINE = MergeTree() table schema and partition key for driver location telemetry.
View Reference Architecture & Worked Solutions (CORE-M36-001)

1. Vitess VSchema Sharding Strategy for OLTP riders and rides Tables

  • Keyspace & Primary Vindex Selection:
    • At (plus during active trips), provision a sharded Vitess keyspace ride_core split into shards (-04, 04-08, , fc-).
    • Shard both riders and rides by rider_id using the xxhash (or binary_md5) Primary Vindex so a rider’s profile, active trip, and payment history are co-located on the exact same MySQL shard, enabling single-shard ACID transactions without two-phase commit:
    • Because drivers and trip-lookup APIs frequently query SELECT * FROM rides WHERE ride_id = ? or WHERE driver_id = ? AND status = 'ACTIVE', define a Consistent Lookup Vindex (ride_id_lookup and driver_active_ride_lookup) so VTGate resolves ride_id -> keyspace_id in without a 64-shard scatter-gather:
{
  "sharded": true,
  "vindexes": {
    "xxhash_vdx": { "type": "xxhash" },
    "ride_id_lookup": {
      "type": "consistent_lookup_unique",
      "params": {
        "table": "ride_core.ride_id_map",
        "from": "ride_id",
        "to": "keyspace_id"
      },
      "owner": "rides"
    }
  },
  "tables": {
    "riders": {
      "column_vindexes": [{ "column": "rider_id", "name": "xxhash_vdx" }]
    },
    "rides": {
      "column_vindexes": [
        { "column": "rider_id", "name": "xxhash_vdx" },
        { "column": "ride_id",  "name": "ride_id_lookup" }
      ]
    }
  }
}

2. Zero-Loss CDC Pipeline from Vitess to ClickHouse (Avoiding Too many parts)

  • Stream Extraction via Vitess VStream:
    • Rather than connecting 64 separate binlog readers directly to underlying MySQL instances (which breaks during Vitess Reshard or MoveTables cutovers), consume the unified Vitess VStream gRPC API via the Debezium Vitess Connector on VTGate. VStream transparently tracks shard splits and emits GTID-ordered row change events into an Apache Kafka topic ride_events_cdc (partitioned by city_id).
  • Micro-Batched ClickHouse Ingestion:
    • Writing directly to ClickHouse triggers DB::Exception: Too many parts. Instead, use ClickHouse’s native Kafka table engine paired with a Materialized View (or the ClickHouse Kafka Connect Sink) configured with stream_flush_interval_ms = 2000 and max_insert_block_size = 1048576:
    • Every , each ClickHouse shard flushes a single sorted bulk block of rows into a ReplacingMergeTree(updated_at_ms) table, keeping part creation at while deduplicating mutable ride state transitions (REQUESTED IN_PROGRESS COMPLETED).

3. ClickHouse MergeTree Schema & Pre-Aggregated Materialized View for Driver Pings / 5 Min

  • Throughput Envelope: .
  • Physical Table & Sparse Primary Key (ORDER BY) Design:
    • Order the MergeTree table from lowest cardinality to highest cardinality matching the surge-pricing query predicates (city_id, geospatial h3_index_r8, 5-minute window, driver_id). Partition by day (toYYYYMMDD(ping_ts)) so each partition holds rows across the cluster (e.g., ClickHouse shards) and old partitions drop cleanly via DROP PARTITION:
CREATE TABLE driver_pings_local ON CLUSTER 'analytics_cluster' (
  ping_ts        DateTime64(3, 'UTC') CODEC(DoubleDelta, ZSTD(1)),
  city_id        UInt16               CODEC(T64, LZ4),
  h3_cell_res8   UInt64               CODEC(T64, LZ4),
  driver_id      UInt64               CODEC(ZSTD(1)),
  lat            Float32              CODEC(Gorilla, LZ4),
  lon            Float32              CODEC(Gorilla, LZ4),
  is_available   UInt8                CODEC(T64, LZ4),
  speed_kph      Float32              CODEC(Gorilla, LZ4)
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/driver_pings', '{replica}')
PARTITION BY toYYYYMMDD(ping_ts)
ORDER BY (city_id, h3_cell_res8, toStartOfFiveMinutes(ping_ts), driver_id)
SETTINGS index_granularity = 8192;

-- Real-Time Surge Pricing Rollup via Incremental Materialized View:
CREATE MATERIALIZED VIEW driver_supply_5m_mv ON CLUSTER 'analytics_cluster'
ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/driver_supply_5m', '{replica}')
PARTITION BY toYYYYMM(window_start)
ORDER BY (city_id, h3_cell_res8, window_start)
AS SELECT
  city_id,
  h3_cell_res8,
  toStartOfFiveMinutes(ping_ts) AS window_start,
  uniqCombined64State(driver_id) AS unique_drivers_state,
  uniqCombined64IfState(driver_id, is_available = 1) AS available_drivers_state,
  avgState(speed_kph) AS avg_speed_state
FROM driver_pings_local
GROUP BY city_id, h3_cell_res8, window_start;
  • Why This Schema Meets Sub-Second SLA: Rather than re-scanning raw rows on every dashboard refresh, driver_supply_5m_mv incrementally updates HyperLogLog (uniqCombined64State) and avgState sketches in memory as each batch is inserted, allowing surge-pricing services to query all H3 hex cells in a city in .

12. Assessment

Review the 4 scenario-based quiz questions at the top of this lesson to test your mastery of Vitess MySQL sharding and ClickHouse columnar vectorized analytics.

13. Summary & Architectural Decision Guide: Workload Family 6 Deep Dive

Analytical (OLAP), Search & Graph Workload Architecture

  • Columnar Layouts & Compression:
    • Store each column in separate files compressed with LZ4 or ZSTD.
    • Columnar layout enables 5x–10x compression ratios and prunes 90%+ of disk I/O on aggregations.
  • Sparse Primary Index Granularity:
    • Sample 1 index mark per 8,192 rows (index_granularity). The entire index sits in RAM.
    • Query filters quickly skip non-matching granules without random disk head seeks.
  • SIMD Vectorized Scans:
    • Process arrays of primitives with AVX-512/NEON instructions in L1/L2 cache, executing billions of row operations per second per CPU core.
  • Distributed Join & Resource Isolation:
    • Avoid large distributed cross-node shuffles; prefer pre-aggregated materialized views or denormalized wide tables.
    • Enforce CPU and memory query quotas (max_memory_usage) to isolate ad-hoc analyst queries from operational real-time dashboards.

14. Authoritative References & Version Context

  • Alexey Milovidov et al. (2016), “ClickHouse: Fast Open-Source Column-Oriented Database Management System”, Yandex / ClickHouse Technical Architecture — clickhouse.com/docs/en/development/architecture
  • Peter Zaitsev and Vadim Tkachenko (2020), “High Performance MySQL: Optimization, Backups, and Replication”, O’Reilly Media — oreilly.com
  • Daniel Abadi, Peter Boncz, and Stavros Harizopoulos (2009), “Column-Oriented Database Systems”, VLDB ‘09 / Foundations and Trends in Databases — doi.org/10.1561/1900000024
  • Version Context: Architecture reflects ClickHouse v24.3 LTS (MergeTree engine, SIMD vectorization) and Vitess v19.0 LTS (VReplication, VTGate distributed query planner).
MASTERY GATE · STEP 71 OF 94

Knowledge check & lesson completion

Score 4 / 4 (100%) on this check to mark this lesson Completed ✓, advance your position on the Master Spine, and add these concepts to your Spaced-Repetition Review Deck.

1. Why does ClickHouse throw the fatal error DB::Exception: Too many parts in all data parts in table when an application performs thousands of individual single-row INSERT statements per second?
2. In Vitess, how does the VTTablet component protect underlying MySQL database instances from catastrophic cascading failure during traffic surges?
3. Why is a columnar database (ClickHouse) orders of magnitude faster than a row-oriented database (MySQL/PostgreSQL) for analytical aggregation queries like SELECT avg(response_time) FROM logs WHERE status = 500?
4. How does Vitess achieve zero-downtime resharding (e.g., splitting Shard -80 into Shards -40 and 40-80) under live production write traffic?
PERSONAL NOTEBOOK

Chapter Notes

Personal engineering insights and observations saved to this browser.

0 notes
Tip: You can also highlight text in the article above to quote directly.

Loading personal notes...