1. FOCUS FIRST · STEP 01 OF 94Thinking in SystemsLearn to treat system design as the engineering of correctness under constraints, not a diagramming exercise.
2. FOCUS NEXT · STEP 02 OF 94Back-of-the-Envelope Estimation0 / 94 Mastered · Pass each 4/4 lesson quiz to advance
8Curriculum Parts
53Chapters & Families
94Substantive Topics
18Production Systems
01 · THINK & MACHINE

Part 1: System Thinking & Machine Fundamentals

Frame requirements, constraints, trade-offs, and capacity math first—then master the physical limits of CPU caches, concurrency, OS I/O, networking, storage media, and single-node saturation.

MILESTONE GATE · PART 01 SYNTHESIS CHECKPOINTSingle-Node Telemetry Gateway at the Saturation Knee
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

You operate a bare-metal telemetry gateway (64 cores, NVMe RAID, 100 Gbps NIC) ingesting 180,000 requests/sec under a 99.95% SLO (p99 < 15 ms). At peak hour, CPU utilization crosses 82%, tail latency spikes to 240 ms, and disk fsync stalls block worker threads.

Decision 1 of 4: Using Little’s Law (L = λW) and M/M/1 queueing dynamics, why did p99 latency jump from 8 ms to 240 ms when CPU utilization rose from 65% to 82%?
Decision 2 of 4: Profiling reveals 38% of CPU cycles are spent in syscall context switches and epoll_wait readiness notifications across 180k req/s. Which kernel I/O architecture eliminates per-operation syscall overhead for both network and NVMe I/O?
Decision 3 of 4: Workers updating shared in-memory telemetry counters suffer severe L3 cache-line bouncing across the 64 cores. Which concurrency design restores linear multi-core scalability?
Decision 4 of 4: To guarantee durability without calling fsync() on every single 512-byte telemetry record (which caps NVMe throughput due to FTL write amplification), how should the persistence path be structured?
02 · BUILDING BLOCKS

Part 2: Architectural & Distributed Building Blocks

Master core systems data structures, spatial and disk indexes, storage engines, distributed clocks, APIs, event logs, caching, replication, consensus, sharding, and resilience.

CHAPTER 095 topics · 230 min

Replication Strategies, Consistency & Consensus

Single-leader, multi-leader, leaderless quorums, chain replication, linearizability, Multi-Paxos, Raft, and BFT.

CHAPTER 104 topics · 180 min

Partitioning, Auto-Sharding & Distributed Transactions

Range/hash sharding, Multi-Raft dynamic range splitting, Two-Phase Commit (2PC), Sagas, TCC, and cross-shard routing.

CHAPTER 114 topics · 195 min

Resilience, Multi-Region & Zero-Trust Foundations

Circuit breakers, adaptive rate limiting, BGP Anycast, Spanner TrueTime/HLC, and Zero-Trust mTLS isolation.

MILESTONE GATE · PART 02 SYNTHESIS CHECKPOINTMulti-Region Distributed Order & Inventory Engine
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

You are designing the core storage, caching, and replication tier for a distributed order service writing 40,000 mutations/sec and serving 300,000 reads/sec across 3 availability zones, requiring zero lost orders and resilience against cache stampedes.

Decision 1 of 4: With 40,000 random order-event writes/sec and point lookups that frequently query non-existent historical keys, which storage engine + filter combination minimizes disk I/O amplification?
Decision 2 of 4: When a hot product catalog key expires in Redis under 50,000 concurrent reads/sec, thousands of workers simultaneously query the database. Which combination prevents this cache stampede?
Decision 3 of 4: After committing an order to the local relational database, the service must publish an OrderCreated event to Kafka without dual-write inconsistency if the process crashes mid-flight. Which pattern guarantees atomic persistence and event publication?
Decision 4 of 4: To prevent split-brain writes when the primary shard leader is isolated by an asymmetric network partition, what invariant must the consensus replication layer enforce?
03 · WORKLOAD PATTERNS

Part 3: Workload Patterns & Distributed Data Engines

Map access shapes and correctness invariants to distributed infrastructure engines across write-heavy commit logs, consensus state machines, stream processors, leaderless rings, transactional SQL, erasure-coded storage, and search.

MILESTONE GATE · PART 03 SYNTHESIS CHECKPOINTHybrid Streaming, Consensus & Petabyte Storage Platform
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

Your infrastructure platform combines a partitioned commit log ingesting 2 GB/s of events, a stream processor computing 1-minute windowed fraud aggregates with late mobile events, an etcd coordination plane, and a 50 PB erasure-coded object archive.

Decision 1 of 4: How does the partitioned commit log (Kafka-style) sustain 2 GB/s of broker egress to multiple consumer groups without saturating user-space memory bandwidth?
Decision 2 of 4: Mobile clients occasionally upload fraud telemetry 45 seconds late due to tunnel connectivity. How should the Flink-style stream processor guarantee accurate 1-minute event-time windows and exactly-once state recovery?
Decision 3 of 4: A worker holding an etcd lease pauses for 12 seconds in a stop-the-world GC cycle while its 5-second lease expires and a new worker acquires the lock. How do you prevent the resumed zombie worker from corrupting shared storage?
Decision 4 of 4: For the 50 PB cold archive requiring 11 nines of durability across 12 failure domains, why is Reed-Solomon (k=8, m=4) erasure coding preferred over 3x replication, and what is its primary trade-off?
04 · DESIGN STUDIO

Part 4: System Design Studio (Design Real Systems)

Independently design end-to-end production application architectures: read-heavy services, real-time messaging, social feeds, search, financial ledgers, and global multi-tenant platforms.

COLLECTION 013 topics · 145 min

Read-Heavy, Object Storage & Media Streaming Systems

URL shorteners, multipart object storage, global CDN byte-range delivery, and adaptive bitrate video streaming pipelines.

COLLECTION 023 topics · 130 min

Real-Time Messaging, Notifications & Collaborative Systems

Multi-channel notification dispatch, WebSocket chat presence, and CRDT/OT real-time collaborative document editing.

COLLECTION 044 topics · 200 min

High-Integrity Ledgers, Flash Sales & Enterprise Platforms

Double-entry ACID payment ledgers, high-concurrency inventory reservations, multi-tenant SaaS isolation, and active-active global routing.

MILESTONE GATE · PART 04 SYNTHESIS CHECKPOINTGlobal Social Commerce, Live Chat & Flash-Sale Platform
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

You are architecting a super-app combining a social timeline (where celebrity accounts have 20M followers), real-time collaborative shopping lists, a flash-sale checkout engine (10,000 items, 500,000 concurrent buyers), and a double-entry payment ledger.

Decision 1 of 4: When a celebrity with 20 million followers posts a flash-sale drop, pure push fan-out-on-write causes a 20M-write queue backlog. How should the timeline architecture handle this skew?
Decision 2 of 4: During the 500,000-user flash sale for 10,000 items, how do you prevent database lock collapse and guarantee zero overselling?
Decision 3 of 4: When a buyer completes payment, the payment service may retry on network timeout. What two mechanisms guarantee financial correctness in the ledger?
Decision 4 of 4: Multiple shoppers edit a shared shopping list concurrently while offline on airplanes and reconnect later. Which data model guarantees deterministic convergence without central lock coordination?
05 · SYSTEMS LIBRARY

Part 5: Systems Library (Anatomy of Real-World Systems)

Deep dives into real-world open-source systems: Envoy, Istio, NGINX, HAProxy, Cassandra, TiDB, CockroachDB, Pulsar, BookKeeper, Flink, ZooKeeper, etcd, Vitess, ClickHouse, Ceph, and HDFS/HBase.

MILESTONE GATE · PART 05 SYNTHESIS CHECKPOINTProduction Open-Source Internals: Envoy, TiKV, Pulsar & ClickHouse
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

You are auditing an open-source stack consisting of Envoy Proxy at the edge, TiDB/TiKV for transactional OLTP, Apache Pulsar + BookKeeper for tiered messaging, and ClickHouse for real-time analytical OLAP.

Decision 1 of 4: How does Envoy Proxy apply routing, cluster membership, and TLS listener updates across thousands of sidecars without dropping active connections or restarting worker threads?
Decision 2 of 4: How does TiDB/TiKV execute distributed ACID transactions across multiple 96 MB Region Raft groups without a single centralized lock manager bottleneck?
Decision 3 of 4: Why can Apache Pulsar add new broker capacity or recover from a broker crash without copying gigabytes of partition data across the network?
Decision 4 of 4: What makes ClickHouse’s MergeTree engine achieve 100x faster analytical scans over 10 billion telemetry rows compared to a row-oriented B-Tree OLTP database?
06 · PRODUCTION

Part 6: Production Engineering (Operate, Debug & Evolve)

Operate real-world distributed systems: observability, telemetry, profiling, incident response, chaos testing, and FinOps.

MILESTONE GATE · PART 06 SYNTHESIS CHECKPOINTSEV-1 Production Incident Triage, eBPF Profiling & Safe Rollout
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

At 02:14 UTC, a canary deployment of your checkout service triggers a 4x p99 latency spike and burns 45% of the monthly error budget in 12 minutes, while traditional application logs show normal handler durations.

Decision 1 of 4: During the first 5 minutes of SEV-1 incident triage, what is the Incident Commander’s primary operational priority?
Decision 2 of 4: OpenTelemetry spans show client-perceived RPC latency is 180 ms, but the server’s application-level span reports only 4 ms. Which kernel-level diagnostic tool isolates where the remaining 176 ms is spent without recompiling the binary?
Decision 3 of 4: Root cause reveals the release also included a migration adding a NOT NULL column with a volatile default on a 400M-row table, acquiring an ACCESS EXCLUSIVE table lock. How must zero-downtime schema migrations be executed?
Decision 4 of 4: To ensure future distributed traces correlate every hop across HTTP gateways, gRPC services, and Kafka consumers without dropping high-error outliers, what telemetry architecture is required?
07 · AI SYSTEMS

Part 7: Distributed AI and ML Systems

Study distributed model training, high-throughput inference serving, GPU orchestration, vector retrieval, and AI platforms.

MILESTONE GATE · PART 07 SYNTHESIS CHECKPOINTDistributed LLM Training & Low-Latency Inference Cluster
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

You lead AI infrastructure for a 70B-parameter LLM platform: pre-training runs across 512 H100 GPUs (8 GPUs/node over NVLink + 400 Gbps RoCEv2), while online serving handles 5,000 concurrent streaming requests alongside a 1B-vector HNSW retrieval index.

Decision 1 of 4: A 70B parameter model in mixed-precision (FP16 weights + FP16 gradients + FP32 Adam states) requires ~1,120 GB of memory—far exceeding a single 80 GB H100 GPU. How should 3D parallelism map onto the 8-GPU NVLink nodes and 400 Gbps inter-node fabric?
Decision 2 of 4: During online LLM serving, naive contiguous KV-cache allocation wastes 65% of GPU HBM on reserved max-sequence-length slots and stalls short requests behind a 32k-token prompt prefill. Which serving architecture resolves both bottlenecks?
Decision 3 of 4: For the 1-billion-vector (1536-dim FP32 ≈ 6 TB raw) RAG retrieval cluster requiring <15 ms p99 recall@10 > 0.95, how do you fit the index economically while maintaining low-latency ANN search?
Decision 4 of 4: During a 512-GPU training job, a single NIC port degrades on Node 37, causing NCCL Ring-AllReduce collectives to hang or drop cluster Model Flops Utilization (MFU) from 46% to 4%. How should the orchestrator & observability plane respond?
08 · CAPSTONES

Part 8: Engineering Capstones (Build and Validate Systems)

Flagship end-to-end engineering capstone projects to prove mastery across architecture, algorithms, and operation.

MILESTONE GATE · PART 08 SYNTHESIS CHECKPOINTPrincipal Capstone Defense: End-to-End Systems Synthesis
4-Decision Scenario (4/4 to earn Badge) ↓
PRODUCTION ARCHITECTURE SCENARIO

In this final Master Spine Capstone defense, you must validate end-to-end architectural invariants across all 4 Flagship Capstones: High-Concurrency Flash-Sale Booking, Distributed Raft KV Store, Multi-GPU LLM Serving, and Enterprise Multi-Tenant RAG.

Decision 1 of 4: In the Flash-Sale Capstone (APP-CAP-001), what happens if a user acquires a Redis inventory reservation token (TTL = 300s) but abandons payment without checking out?
Decision 2 of 4: In the Distributed KV Store Capstone (INF-CAP-001), how does a Raft-backed LSM storage node serve linearizable point reads without paying the latency of appending a full Raft log entry on every GET?
Decision 3 of 4: In the Distributed LLM Serving Capstone (AI-CAP-001), when GPU KV-cache block utilization reaches 98% mid-generation across 200 active sequences, how does the scheduler prevent an Out-Of-Memory (OOM) crash?
Decision 4 of 4: In the Enterprise Multi-Tenant RAG Capstone (AI-CAP-002), how do you guarantee both high retrieval precision on domain-specific acronyms/IDs and strict zero cross-tenant data leakage?
CROSS-CUTTING FIELD GUIDES

Production Systems Library

Architecture field guides for 18 real-world distributed systems, cross-linked back into the theoretical foundation.