Modern software systems rarely fail because hardware is too slow; they fail because software wastes orders of magnitude of hardware capability through unmonitored lock contention, memory allocation thrashing, cache misses, and hidden queueing delay. When a microservice consumes 90% CPU but produces low throughput, running top tells you nothing about why the CPU is spinning.
Performance engineering is the empirical discipline of identifying, measuring, and eliminating bottlenecks across hardware, operating system kernels, and user-space runtimes:
- CPU & Memory Profiling: Using statistical sampling profilers (
perf) and interactive Flame Graphs to differentiate between active compute execution (On-CPU) and thread descheduling waiting on locks or disk I/O (Off-CPU). - Extended Berkeley Packet Filter (eBPF): Running sandboxed programs inside the Linux kernel to trace socket lifecycles, kernel run-queue scheduler delays (
runqlat), and file descriptor exhaustion with negligible (<1%) overhead. - Queueing Theory & Tail Latency: Applying Kingman’s formula to model the catastrophic non-linear latency hockey stick that occurs as server utilization exceeds 75–80%.
This module equips systems architects with the mathematical models and kernel-level instrumentation necessary to extract peak performance and squash tail-latency anomalies.