DISTRIBUTED AI AND ML SYSTEMS / C05

AI Inference Serving & Engine Architecture

Deconstruct production LLM inference engines: continuous batching (iteration-level scheduling), PagedAttention virtual memory KV caches, chunked prefill, and speculative decoding.

1 topic360 min estimatedPublished

Serving Large Language Models (LLMs) in production environments presents a radical departure from traditional stateless web microservices. Autoregressive text generation is memory-bandwidth bound during token generation (Decode phase) and compute-bound during prompt ingestion (Prefill phase).

Naive inference systems that batch requests statically suffer from the “curse of variable output lengths”: an entire GPU batch stalls, wasting up to 80% of its compute cycles, waiting for the single longest response to finish. Furthermore, pre-allocating contiguous memory buffers for the Key-Value (KV) cache wastes over 60% of expensive GPU High-Bandwidth Memory (HBM) due to internal and external fragmentation.

This module deconstructs modern LLM serving engines (vLLM, TensorRT-LLM, TGI): continuous iteration-level scheduling, PagedAttention virtual memory mapping, chunked prefill scheduling, speculative decoding, and production serving SLAs.

What you will learn

  • Compare naive static batching versus continuous iteration-level batching (Orca/vLLM) to maximize GPU tensor core occupancy
  • Analyze KV cache memory fragmentation and deconstruct PagedAttention block tables inspired by OS virtual memory pages
  • Design chunked prefill and piggyback scheduling to prevent prefill compute bursts from starving ongoing decode token latency
  • Implement speculative decoding pipelines evaluating draft models against target verifier models for 2x-3x latency reduction
  • Quantify time-to-first-token (TTFT) and inter-token-latency (ITL) trade-offs across auto-scaling inference cluster topologies
STAGE TOPICS

Topics in this chapter

Step 84High-Throughput LLM Inference Serving: Continuous Batching, PagedAttention, and Chunked PrefillDeconstruct production LLM inference engines: continuous batching (iteration-level scheduling), PagedAttention virtual memory KV caches, chunked prefill, and speculative decoding.55 min ↗