DISTRIBUTED AI AND ML SYSTEMS / C03

LLM Inference Mechanics: Prefill, Decode & KV Cache

Deconstruct autoregressive Transformer inference across GPU accelerators: compute-bound Prefill vs. memory-bandwidth-bound Decode, Roofline arithmetic intensity, KV cache sizing, Multi-Query / Grouped-Query Attention (MQA/GQA), and Speculative Decoding.

1 topic240 min estimatedPublished

Serving Large Language Models (LLMs) at low Time-To-First-Token (TTFT) and low Inter-Token Latency (ITL) is fundamentally a hardware systems problem governed by the GPU Roofline model and High-Bandwidth Memory (HBM) capacity. Autoregressive generation splits into two physically distinct regimes: Prefill, which processes all prompt tokens in parallel and saturates Tensor Core FLOPs, and Decode, which generates one token per step and is bottlenecked by streaming gigabytes of weights and Key-Value (KV) cache states from HBM into SRAM.

This chapter deconstructs the core mechanics of LLM inference engines before scaling to multi-tenant serving in AI-C05-001: Roofline arithmetic intensity analysis, exact KV cache memory sizing for 70B+ parameter models, Grouped-Query Attention (GQA) compression, SRAM-tiled FlashAttention, and Speculative Decoding.

What you will learn

  • Distinguish the compute-bound Prefill phase (parallel matrix-matrix multiplication) from the memory-bandwidth-bound Decode phase (sequential matrix-vector generation)
  • Derive Roofline arithmetic intensity ridge points () on H100 GPUs and explain why naive decode underutilizes Tensor Cores
  • Calculate exact Key-Value (KV) cache VRAM footprints across Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Multi-Query Attention (MQA)
  • Evaluate FP8/INT4 KV cache quantization and FlashAttention SRAM tiling to eliminate HBM bandwidth bottlenecks
  • Formulate Speculative Decoding speedup Equations using small draft models and parallel target-model verification
STAGE TOPICS

Topics in this chapter

Step 83LLM Inference: Prefill, Decode and KV CacheMaster the mechanics of autoregressive LLM serving, KV cache memory sizing, PagedAttention, and continuous batching.30 min ↗