Capstone Project: Distributed LLM Inference Serving Platform

Serving large language models (such as LLaMA-3 70B/405B, DeepSeek-V3, and Mixtral MoE) at production scale presents fundamentally different systems challenges than traditional microservices or distributed relational databases.

LLM inference is characterized by two distinct computational phases with opposing bottlenecks:

1. Prefill (Prompt Processing): Highly compute-bound matrix-matrix multiplications ($GEMM$) achieving near-peak tensor core teraFLOPS.
2. Decode (Autoregressive Generation): Severely memory-bandwidth-bound matrix-vector multiplications ($GEMV$), where the entire multi-gigabyte Key-Value (KV) cache must be reloaded from High Bandwidth Memory (HBM) to generate each single new token.

In this Capstone, learners architect a complete distributed serving platform (modeled after vLLM, TensorRT-LLM, and Triton Inference Server) that solves KV cache fragmentation, coordinates multi-GPU tensor communication, and deploys speculative decoding to accelerate token generation.