1. FOCUS FIRST · STEP 01 OF 94 Thinking in Systems Learn to treat system design as the engineering of correctness under constraints, not a diagramming exercise.
2. FOCUS NEXT · STEP 02 OF 94 Back-of-the-Envelope Estimation 0 / 94 Mastered · Pass each 4/4 lesson quiz to advance
8 Curriculum Parts
53 Chapters & Families
94 Substantive Topics
18 Production Systems
01 · THINK & MACHINE
Part 1: System Thinking & Machine Fundamentals Frame requirements, constraints, trade-offs, and capacity math first—then master the physical limits of CPU caches, concurrency, OS I/O, networking, storage media, and single-node saturation.
Problem framing, functional vs non-functional requirements, napkin math, and the 6-step engineering design sequence.
Availability error budgets, Zipfian skew, fan-out amplification, and Architecture Decision Records (ADRs).
CPU caches, L1/L2/L3/DRAM latency hierarchy, NUMA topologies, and the physical memory wall.
Hardware memory ordering (TSO vs ARM64), MESI false sharing, CAS loops, ABA mitigation, and hazard pointers.
Non-blocking epoll readiness queues, Linux io_uring submission/completion rings, DPDK, and AF_XDP.
NVMe SSDs, FTL write amplification, kernel page cache writeback, and fsync durability guarantees.
L4 vs L7 load balancing, Direct Server Return (DSR), TLS 1.3 termination, reverse proxies, and API gateways.
Measure before scaling: Little’s Law, queueing delay, USL saturation formulas, and the saturation knee.
MILESTONE GATE · PART 01 SYNTHESIS CHECKPOINT Single-Node Telemetry Gateway at the Saturation Knee
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO You operate a bare-metal telemetry gateway (64 cores, NVMe RAID, 100 Gbps NIC) ingesting 180,000 requests/sec under a 99.95% SLO (p99 < 15 ms). At peak hour, CPU utilization crosses 82%, tail latency spikes to 240 ms, and disk fsync stalls block worker threads.
🏆 Part 01 Verified: Hardware & Single-Node Architect Part 01 synthesis milestone saved to your learning log and study portfolio!
02 · BUILDING BLOCKS
Part 2: Architectural & Distributed Building Blocks Master core systems data structures, spatial and disk indexes, storage engines, distributed clocks, APIs, event logs, caching, replication, consensus, sharding, and resilience.
B+ trees, spatial indexes (Geohash/H3/S2/R-trees), LSM-trees, Bloom filters, consistent hashing, and sketches.
Buffer pool eviction, WAL/ARIES recovery, MVCC snapshot visibility, and transaction isolation anomalies.
Fallacies of distributed computing, network timeouts, idempotency, physical clock drift, Lamport timestamps, and vector clocks.
gRPC, Protocol Buffers wire encoding, schema evolution, versioning, and backward compatibility.
Dual-write race mitigation, transactional outbox, Debezium WAL Change Data Capture (CDC), and consumer idempotency.
Cache-aside, write-through, CDC invalidation, singleflight coalescing, and XFetch probabilistic early expiration.
Event time vs processing time, bounded-out-of-orderness watermarks, windowing, and Chandy-Lamport snapshots.
Clean architecture boundaries, CQRS read/write separation, and fault-isolated Shuffle-Sharding cell topologies.
Single-leader, multi-leader, leaderless quorums, chain replication, linearizability, Multi-Paxos, Raft, and BFT.
Range/hash sharding, Multi-Raft dynamic range splitting, Two-Phase Commit (2PC), Sagas, TCC, and cross-shard routing.
Circuit breakers, adaptive rate limiting, BGP Anycast, Spanner TrueTime/HLC, and Zero-Trust mTLS isolation.
MILESTONE GATE · PART 02 SYNTHESIS CHECKPOINT Multi-Region Distributed Order & Inventory Engine
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO You are designing the core storage, caching, and replication tier for a distributed order service writing 40,000 mutations/sec and serving 300,000 reads/sec across 3 availability zones, requiring zero lost orders and resilience against cache stampedes.
🏆 Part 02 Verified: Distributed Building Blocks Architect Part 02 synthesis milestone saved to your learning log and study portfolio!
03 · WORKLOAD PATTERNS
Part 3: Workload Patterns & Distributed Data Engines Map access shapes and correctness invariants to distributed infrastructure engines across write-heavy commit logs, consensus state machines, stream processors, leaderless rings, transactional SQL, erasure-coded storage, and search.
Partitioned append-only logs, zero-copy transfers, consumer offsets, event-time watermarks, and Chandy-Lamport checkpoints.
Formal Raft state machines, leader election safety, log replication, time-bounded leases, and monotonic fencing tokens.
Decentralized Dynamo rings, sloppy quorums, Merkle anti-entropy, and Multi-Raft distributed SQL transactions.
Reed-Solomon erasure coding over Galois fields, Ceph CRUSH placement, distributed inverted indexes, and columnar OLAP.
MILESTONE GATE · PART 03 SYNTHESIS CHECKPOINT Hybrid Streaming, Consensus & Petabyte Storage Platform
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO Your infrastructure platform combines a partitioned commit log ingesting 2 GB/s of events, a stream processor computing 1-minute windowed fraud aggregates with late mobile events, an etcd coordination plane, and a 50 PB erasure-coded object archive.
Decision 1 of 4: How does the partitioned commit log (Kafka-style) sustain 2 GB/s of broker egress to multiple consumer groups without saturating user-space memory bandwidth? Deserializing every message batch into JVM heap objects, re-validating JSON schemas, and re-compressing payloads per consumer socket. Storing all topic messages inside a single global B+Tree indexed by consumer group UUID and executing random 4 KB page reads. Appending to immutable segment files and transferring bytes directly from OS page cache to NIC buffers via sendfile() zero-copy DMA. Broadcasting unbuffered records over unreliable UDP multicast datagrams without broker-side disk persistence or offset tracking.
Decision 2 of 4: Mobile clients occasionally upload fraud telemetry 45 seconds late due to tunnel connectivity. How should the Flink-style stream processor guarantee accurate 1-minute event-time windows and exactly-once state recovery? Using broker processing-time windows and silently discarding any mobile event whose payload timestamp differs from the wall clock by more than 100 ms. Halting all TaskManager operator threads across the streaming cluster for 45 seconds at the end of every minute before emitting window results. Re-reading the entire 30-day Kafka retention log from offset 0 into heap memory whenever a late mobile event arrives after window closure. Tracking event-time progress with bounded-out-of-orderness watermarks + allowed lateness side outputs, backed by asynchronous Chandy-Lamport barrier snapshots to durable storage.
Decision 3 of 4: A worker holding an etcd lease pauses for 12 seconds in a stop-the-world GC cycle while its 5-second lease expires and a new worker acquires the lock. How do you prevent the resumed zombie worker from corrupting shared storage? Attaching a monotonically increasing fencing token (such as the etcd Raft ModRevision) to every storage write and rejecting any write whose token is lower than the storage server’s highest seen token. Having the worker inspect its own cached in-memory boolean isLeader flag inside a local mutex before issuing the storage write RPC. Having the new lock holder send an asynchronous HTTP POST webhook with a polite SIGTERM request to the paused worker’s IP address. Increasing the etcd lease TTL to 24 hours across all workers so stop-the-world JVM garbage collection pauses never expire active locks.
Decision 4 of 4: For the 50 PB cold archive requiring 11 nines of durability across 12 failure domains, why is Reed-Solomon (k=8, m=4) erasure coding preferred over 3x replication, and what is its primary trade-off? RS(8,4) requires zero cross-rack network hops during node rebuilds and reduces small 4 KB random write tail latency by 10x compared to 3x replication. RS(8,4) reduces raw storage overhead from 3.0x to 1.5x while tolerating any 4 concurrent domain failures, at the cost of higher CPU/network reconstruction amplification during degraded reads and repairs. RS(8,4) co-locates all 8 data chunks and 4 parity chunks onto a single NVMe drive to eliminate inter-node network bandwidth consumption completely. RS(8,4) can only tolerate a single disk failure across the 12 domains whereas 3x replication tolerates up to 4 simultaneous rack failures.
🏆 Part 03 Verified: Workload Engines Architect Part 03 synthesis milestone saved to your learning log and study portfolio!
04 · DESIGN STUDIO
Part 4: System Design Studio (Design Real Systems) Independently design end-to-end production application architectures: read-heavy services, real-time messaging, social feeds, search, financial ledgers, and global multi-tenant platforms.
URL shorteners, multipart object storage, global CDN byte-range delivery, and adaptive bitrate video streaming pipelines.
Multi-channel notification dispatch, WebSocket chat presence, and CRDT/OT real-time collaborative document editing.
Hybrid push/pull timeline fan-out under celebrity skew and low-latency FST prefix autocomplete with geospatial indexing.
Double-entry ACID payment ledgers, high-concurrency inventory reservations, multi-tenant SaaS isolation, and active-active global routing.
MILESTONE GATE · PART 04 SYNTHESIS CHECKPOINT Global Social Commerce, Live Chat & Flash-Sale Platform
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO You are architecting a super-app combining a social timeline (where celebrity accounts have 20M followers), real-time collaborative shopping lists, a flash-sale checkout engine (10,000 items, 500,000 concurrent buyers), and a double-entry payment ledger.
Decision 1 of 4: When a celebrity with 20 million followers posts a flash-sale drop, pure push fan-out-on-write causes a 20M-write queue backlog. How should the timeline architecture handle this skew? Use a hybrid fan-out architecture: push fan-out-on-write for normal users (<= 25,000 followers) and pull fan-out-on-read (merging celebrity outboxes at read time) for high-follower accounts. Silently drop timeline posts from verified accounts with more than 100,000 followers whenever Kafka consumer lag exceeds 5 seconds during peak hours. Store the entire 500-million-user social follow graph in a single unsharded relational table enforced with strict SERIALIZABLE two-phase locking. Require all 20 million followers’ mobile clients to poll an uncached multi-way SQL JOIN query against the primary database every 500 ms.
Decision 2 of 4: During the 500,000-user flash sale for 10,000 items, how do you prevent database lock collapse and guarantee zero overselling? Execute an unisolated SELECT count(*) FROM orders followed by an application-level INSERT without row locks so database queries complete faster. Acquire an exclusive LOCK TABLES inventory WRITE table lock across the primary relational database for the entire 10-minute duration of the flash sale. Gate admissions at the edge, decrement inventory atomically in Redis (Lua script) to issue time-bounded reservation tokens, and drain reservations asynchronously into the database. Accept all 500,000 checkout payments optimistically at the edge gateway and asynchronously refund 98% of over-allocated orders via batch email three days later.
Decision 3 of 4: When a buyer completes payment, the payment service may retry on network timeout. What two mechanisms guarantee financial correctness in the ledger? Mutating a single IEEE-754 floating-point balance column in place (UPDATE accounts SET balance = balance - amount) over idempotent HTTP GET requests. Relying on eventual consistency with asynchronous Last-Write-Wins (LWW) conflict resolution on mutable account balance registers across regions. Caching authoritative account balances exclusively in client browser localStorage and synchronizing batch totals to the server once every midnight. Enforcing client-generated idempotency keys (unique constraint on idempotency_key) paired with immutable double-entry ledger postings (sum of debits + credits = 0).
Decision 4 of 4: Multiple shoppers edit a shared shopping list concurrently while offline on airplanes and reconnect later. Which data model guarantees deterministic convergence without central lock coordination? Overwriting the entire shared shopping list document in the database with whichever mobile client reconnects last using local device wall-clock timestamps. State-based or operation-based Conflict-Free Replicated Data Types (CRDTs, such as Observed-Remove Set with causal metadata) whose merge function is associative, commutative, and idempotent. Enforcing a synchronous blocking Two-Phase Commit (2PC) lock across all participating mobile devices before allowing any offline local keystroke to render. Rejecting and discarding all offline shopping list edits whenever a client’s round-trip network latency to the primary region exceeds 50 ms.
🏆 Part 04 Verified: System Design Studio Architect Part 04 synthesis milestone saved to your learning log and study portfolio!
05 · SYSTEMS LIBRARY
Part 5: Systems Library (Anatomy of Real-World Systems) Deep dives into real-world open-source systems: Envoy, Istio, NGINX, HAProxy, Cassandra, TiDB, CockroachDB, Pulsar, BookKeeper, Flink, ZooKeeper, etcd, Vitess, ClickHouse, Ceph, and HDFS/HBase.
Envoy Proxy dynamic xDS, Istio service mesh, NGINX event loop, and HAProxy/Kong edge gateways.
Apache Cassandra Dynamo rings, TiDB/TiKV Percolator 2PC, and CockroachDB HLC consensus.
Apache Pulsar decoupled BookKeeper storage ledgers, quorum striping, and tiered object storage.
Apache Flink asynchronous Chandy-Lamport barrier checkpoints and exactly-once processing guarantees.
Apache ZooKeeper Zab protocol and etcd Raft consensus with MVCC bbolt storage.
Vitess MySQL sharding, ClickHouse columnar analytics, Ceph CRUSH mapping, and HDFS/HBase architecture.
MILESTONE GATE · PART 05 SYNTHESIS CHECKPOINT Production Open-Source Internals: Envoy, TiKV, Pulsar & ClickHouse
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO You are auditing an open-source stack consisting of Envoy Proxy at the edge, TiDB/TiKV for transactional OLTP, Apache Pulsar + BookKeeper for tiered messaging, and ClickHouse for real-time analytical OLAP.
🏆 Part 05 Verified: Open-Source Systems Specialist Part 05 synthesis milestone saved to your learning log and study portfolio!
06 · PRODUCTION
Part 6: Production Engineering (Operate, Debug & Evolve) Operate real-world distributed systems: observability, telemetry, profiling, incident response, chaos testing, and FinOps.
OpenTelemetry standards, distributed trace propagation, metrics collection, and correlation IDs.
Systematic debugging under fire, timeline reconstruction, and blameless postmortem analysis.
Fault injection, Jepsen partition testing, network jitter simulation, and invariant verification.
Canary deployments, blue-green cutovers, online zero-downtime database schema migrations, and backup verification.
CPU flame graphs, memory allocation bottlenecks, off-CPU analysis, and non-invasive kernel tracing with eBPF.
Hardware cost models, bandwidth egress budgeting, multi-tier storage lifecycle, and cluster utilization tuning.
Automated node drain, live partition rebalancing, dynamic decommissioning, and automated repair runbooks.
MILESTONE GATE · PART 06 SYNTHESIS CHECKPOINT SEV-1 Production Incident Triage, eBPF Profiling & Safe Rollout
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO At 02:14 UTC, a canary deployment of your checkout service triggers a 4x p99 latency spike and burns 45% of the monthly error budget in 12 minutes, while traditional application logs show normal handler durations.
Decision 1 of 4: During the first 5 minutes of SEV-1 incident triage, what is the Incident Commander’s primary operational priority? Spend 2 hours bisecting git commits and authoring a 10-page root-cause postmortem document before altering any live production state. Restart all stateful database cluster nodes simultaneously via hard power-off to clear potentially fragmented OS page caches. Disable OpenTelemetry distributed tracing and Prometheus metric scraping across all production pods to save 1% CPU overhead. Mitigate customer impact immediately (halt/rollback the canary, shed excess load, or shift traffic) while preserving forensic telemetry for post-mitigation root-cause analysis.
Decision 2 of 4: OpenTelemetry spans show client-perceived RPC latency is 180 ms, but the server’s application-level span reports only 4 ms. Which kernel-level diagnostic tool isolates where the remaining 176 ms is spent without recompiling the binary? Adding synchronous console.log() debug print statements to every function in the codebase and deploying a new production build. Attaching Linux eBPF kprobes/tracepoints (e.g., off-CPU scheduler wakeups, TCP retransmits, and socket queue dwell time) alongside CPU/off-CPU Flame Graphs. Running an ICMP ping 127.0.0.1 loopback script in a tight bash loop on the bastion host to test local network card latency. Dropping all secondary indexes on the production relational database during peak traffic to test whether query plans change.
Decision 3 of 4: Root cause reveals the release also included a migration adding a NOT NULL column with a volatile default on a 400M-row table, acquiring an ACCESS EXCLUSIVE table lock. How must zero-downtime schema migrations be executed? Using the Expand-and-Contract pattern: (1) add nullable column metadata-only, (2) dual-write in code & backfill in small throttled batches, (3) validate via NOT VALID constraint + concurrent validation, and (4) switch reads/contract. Executing ALTER TABLE ... ADD COLUMN ... NOT NULL directly during Tuesday noon peak traffic with lock_timeout = 0 so the migration waits indefinitely for locks. Exporting all 400 million rows to local CSV files, dropping the production table, recreating it with the new schema, and re-importing over 6 hours. Permanently disabling all database foreign keys, check constraints, and transaction isolation levels across the production cluster.
Decision 4 of 4: To ensure future distributed traces correlate every hop across HTTP gateways, gRPC services, and Kafka consumers without dropping high-error outliers, what telemetry architecture is required? Generating a fresh disconnected random trace_id on every microservice hop and applying 0.01% head-based uniform random sampling at the edge. Writing raw unredacted credit-card request payloads into unstructured local text files on ephemeral container overlay filesystems. Propagating W3C Trace Context (traceparent) headers across all sync/async boundaries and using tail-based sampling in the collector to retain 100% of error and high-latency traces. Configuring head-based sampling to record 100% of synthetic /healthz liveness probes while dropping 99.9% of checkout API requests.
🏆 Part 06 Verified: Production SRE & Operations Lead Part 06 synthesis milestone saved to your learning log and study portfolio!
07 · AI SYSTEMS
Part 7: Distributed AI and ML Systems Study distributed model training, high-throughput inference serving, GPU orchestration, vector retrieval, and AI platforms.
Accelerator microarchitecture, HBM memory hierarchies, ZeRO memory sharding, and FSDP.
Ring-AllReduce algorithms, NVLink vs InfiniBand, RDMA, and NCCL communication primitives.
Prefill vs decode disaggregation, KV cache sizing, PagedAttention, and continuous batching.
Hierarchical Navigable Small World (HNSW), Inverted File (IVF), and Product Quantization (PQ).
All-or-nothing gang scheduling, topology-aware placement, multi-node GPU fault recovery, and Kueue/Ray.
Point-in-time correct joins, streaming vector indexing, and embedding cache architectures.
Multi-model actor-critic orchestration (PPO), Direct Preference Optimization (DPO), and Ray workflows.
Durable execution DAGs, context sharding, tool calling execution engines, and multimodal streaming.
Model Flops Utilization (MFU), TTFT/ITL latency distributions, GPU kernel profiling, and NCCL deadlocks.
MILESTONE GATE · PART 07 SYNTHESIS CHECKPOINT Distributed LLM Training & Low-Latency Inference Cluster
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO You lead AI infrastructure for a 70B-parameter LLM platform: pre-training runs across 512 H100 GPUs (8 GPUs/node over NVLink + 400 Gbps RoCEv2), while online serving handles 5,000 concurrent streaming requests alongside a 1B-vector HNSW retrieval index.
Decision 1 of 4: A 70B parameter model in mixed-precision (FP16 weights + FP16 gradients + FP32 Adam states) requires ~1,120 GB of memory—far exceeding a single 80 GB H100 GPU. How should 3D parallelism map onto the 8-GPU NVLink nodes and 400 Gbps inter-node fabric? Place communication-intensive Tensor Parallelism (TP=8) across separate racks over oversubscribed WAN links and keep Data Parallelism inside a single GPU. Store all 1,120 GB of model weights and optimizer states on host CPU swap disks and stream every transformer layer over PCIe on each micro-batch. Keep communication-heavy Tensor Parallelism (TP=8) within each 900 GB/s NVLink node, and place Pipeline Parallelism (PP) and ZeRO/FSDP Data Parallelism across the inter-node RoCEv2 network. Train 512 completely independent copies of the 70B model with zero gradient synchronization across nodes until the final training epoch completes.
Decision 2 of 4: During online LLM serving, naive contiguous KV-cache allocation wastes 65% of GPU HBM on reserved max-sequence-length slots and stalls short requests behind a 32k-token prompt prefill. Which serving architecture resolves both bottlenecks? PagedAttention (non-contiguous virtual-to-physical KV cache blocks) combined with iteration-level continuous batching and chunked/disaggregated prefill-decode scheduling. Static request batching with batch_size=1 while pre-allocating a contiguous 32,768-token FP32 KV buffer in HBM for every incoming connection. Discarding the KV cache entirely and recomputing all historical token Key/Value tensors from scratch via full-prompt forward passes on every generated token. Offloading the memory-bandwidth-bound autoregressive decode phase onto a single-core host CPU over PCIe to conserve GPU HBM capacity.
Decision 3 of 4: For the 1-billion-vector (1536-dim FP32 ≈ 6 TB raw) RAG retrieval cluster requiring <15 ms p99 recall@10 > 0.95, how do you fit the index economically while maintaining low-latency ANN search? Execute brute-force exact L2/cosine distance scans over all 6 TB of uncompressed FP32 vectors across network-attached spinning hard drives on every query. Combine HNSW / IVF graph-partitioned indexing with Product Quantization (PQ / SQ8) compression in RAM and re-rank the top-K candidates using full-precision vectors from NVMe. Hash every 1,536-dimensional embedding vector into a single 8-bit integer bucket and execute standard SQL B-Tree range scans. Rebuild the entire 1-billion-vector HNSW hierarchical proximity graph from scratch synchronously on every single document insert.
Decision 4 of 4: During a 512-GPU training job, a single NIC port degrades on Node 37, causing NCCL Ring-AllReduce collectives to hang or drop cluster Model Flops Utilization (MFU) from 46% to 4%. How should the orchestrator & observability plane respond? Wait 72 hours for the default Linux kernel TCP keepalive timeout to expire before alerting an on-call infrastructure engineer. Continue training while silently dropping all gradient tensors from Node 37 on every backward pass without adjusting the optimizer step. Terminate the entire 512-GPU job and restart pre-training from step 0 with random weight initialization because GPU state cannot be checkpointed. Monitor per-rank NCCL flight-recorder telemetry & step-time straggler skew, trigger automated health-check cordoning of the degraded node, and resume from the latest asynchronous distributed checkpoint on a healthy standby node.
🏆 Part 07 Verified: Distributed AI Systems Architect Part 07 synthesis milestone saved to your learning log and study portfolio!
08 · CAPSTONES
Part 8: Engineering Capstones (Build and Validate Systems) Flagship end-to-end engineering capstone projects to prove mastery across architecture, algorithms, and operation.
Build an end-to-end reservation engine capable of surviving 50x bursts with zero overselling.
Build a replicated, sharded key-value store with LSM-tree storage, Raft consensus, and failure detection.
Build a multi-GPU LLM inference server featuring continuous batching, PagedAttention, and KV cache eviction.
Build a production-grade multi-tenant RAG platform with hybrid dense/sparse search and streaming generation.
MILESTONE GATE · PART 08 SYNTHESIS CHECKPOINT Principal Capstone Defense: End-to-End Systems Synthesis
4-Decision Scenario (4/4 to earn Badge) ↓ PRODUCTION ARCHITECTURE SCENARIO In this final Master Spine Capstone defense, you must validate end-to-end architectural invariants across all 4 Flagship Capstones: High-Concurrency Flash-Sale Booking, Distributed Raft KV Store, Multi-GPU LLM Serving, and Enterprise Multi-Tenant RAG.
🏆 Part 08 Verified: Principal Distributed Systems Architect Part 08 synthesis milestone saved to your learning log and study portfolio!
CROSS-CUTTING FIELD GUIDES
Production Systems Library Architecture field guides for 18 real-world distributed systems, cross-linked back into the theoretical foundation.