Systems Benchmarking, the Saturation Knee & Vertical vs. Horizontal Scaling Decisions

Engineers frequently treat scaling as an ideological mandate: “Cloud-native systems must always scale horizontally.” This mindset leads teams to replace simple, reliable single-server systems with dozens of microservices, distributed databases, and complex service meshes—only to suffer higher latency, worse reliability, and massive cloud bills.

A disciplined systems architect does not scale based on hype; they scale based on empirical measurement.

This lesson explores the science of performance benchmarking, how to eliminate the fatal Coordinated Omission flaw in load testing, how to model cluster scaling limits with the Universal Scalability Law (USL), and how to rigorously evaluate vertical scale-up vs. horizontal scale-out.

1. Problem and Motivation

Consider an engineering team preparing for a major product launch:

1. They run ApacheBench (ab -n 100000 -c 100) from their development laptop against the staging server.
2. The benchmark reports: Throughput: 8,500 req/sec, 99th percentile latency: 8.5 ms. The team celebrates and approves the architecture for production.
3. On launch day, real users flood the site at $6,000\text{ req/sec}$. Within ten minutes, user requests time out, the payment gateway backs up, and the site crashes. Real p99 latency was not $8.5\text{ ms}$; it was $14\text{ seconds}$!

What went wrong?

• Coordinated Omission: ApacheBench used closed-loop request generation. When the server suffered periodic $5\text{-second}$ garbage collection pauses, the benchmark stopped generating requests, omitting the millions of requests that would have backed up in production.
• Shared Resource Contention: The team tested against an unrepresentative environment with zero network latency.
• Unmodeled Coherence Overhead: When the team panicked and spun up 20 additional server instances, database lock contention and cache invalidation storms caused total system throughput to drop—a phenomenon known as retrograde scalability.

2. Mental Model & Architectural Overview

SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED SYSTEMS

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. The Closed-Loop Trap (wrk, ab, jmeter default) & Open-Loop Testing (wrk2, k6 constant-arrival)

Establishes the ingress and structural boundary across The Closed-Loop Trap (wrk, ab, jmeter default) & Open-Loop Testing (wrk2, k6 constant-arrival): Client sends request, WAITS for response before sending next; If server stalls for 5s (e.g. GC pause), client also stalls!; Only 1 stalled request is recorded; 10,000 intended requests are never generated! p99 appears falsely low.

Architectural Invariant: Latency is measured from SCHEDULED arrival to completion.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. Tooling Standard: wrk2, k6, vegeta, hey, Linux perf

Coordinates core data-plane execution and state transitions through Tooling Standard: wrk2, k6, vegeta, hey, Linux perf: Always run load generator on a separate machine across dedicated 10G link.

Architectural Invariant: Always run load generator on a separate machine across dedicated 10G link.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Scaling Decision Thresholds & Benchmark before scaling: closed-loop testing lies about p99 latency; the Universal Scalability Law proves that uncoordinated clustering leads to retrograde collapse.

Enforces durability, state convergence, and output verification across Scaling Decision Thresholds & Benchmark before scaling: closed-loop testing lies about p99 latency; the Universal Scalability Law proves that uncoordinated clustering leads to retrograde collapse.: 1. Vertical Scaling (Scale-Up); Always exhaust first: 16 core → 64 core is immediate zero-ops; No distributed consensus, no cross-node network hops, lowest cost.

Architectural Invariant: Watch for coherency penalty (κ): adding nodes can REDUCE throughput!.

The Three Scalability Regimes (USL)

Throughput X(N)
      ▲
      │                 Ideal Linear Scaling (X(N) = γN)
      │               . '
      │             . '     Contention Bound (Amdahl's Law: σ > 0)
      │           . '    . - - - - - - - - - - - - - - - - -
      │         . '   . '
      │       . '   .'
      │     . '   .'     Peak Throughput N*
      │   . '   .'            │
      │ . '   .'              ▼
      │.    .'             \  \
      │   .'                \  \  Retrograde Scalability (Coherency: κ > 0)
      │ .'                   \  \  (Adding nodes REDUCES net throughput!)
      └────────────────────────────► Number of Nodes / Cores (N)

3. Core Principles & Mathematical Formulations

3.1 The Coordinated Omission Flaw

In closed-loop benchmarking: $$\text{Client Action:}{\text{Send Req}}_i⟶{\text{Wait for Resp}}_i⟶{\text{Send Req}}_{i+1}$$

If the server freezes for $10\text{ seconds}$ at $t=50$:

• The closed-loop client sits blocked waiting for the single active request.
• The benchmark records: 1 request taking $10\text{ seconds}$.
• The thousands of requests that should have arrived during those 10 seconds are omitted from the test dataset entirely!

In open-loop benchmarking: $$\text{Client Action:}{\text{Schedule Req}}_i\text{ at }t=i\cdot \frac{1}{\lambda }$$

Requests are emitted at an independent arrival rate $\lambda$. Latency is measured as: $$\text{Latency}=\text{Completion Time}-\text{Scheduled Arrival Time}$$

If the server freezes for $10\text{ seconds}$, all scheduled requests continue piling up in the virtual queue, and the open-loop tool accurately records that thousands of requests suffered multi-second queuing delays.

3.2 The Universal Scalability Law (USL)

Amdahl’s Law models speedup under serialized contention ($\sigma$). However, real distributed systems also suffer from coherency penalty ($\kappa$)—the overhead of cross-node gossip, distributed transactions, cache invalidation messages, and two-phase commits, which scale as $O(N^2)$.

Dr. Neil Gunther synthesized these dynamics into the Universal Scalability Law (USL):

Where:

• $N$ is the number of concurrent worker processes, CPU cores, or server nodes.
• $\gamma$ is the throughput of a single standalone worker under zero contention.
• $\sigma$ (sigma) is the contention parameter ($0\le \sigma \le 1$), representing serialized non-parallelizable work (Amdahl bottleneck).
• $\kappa$ (kappa) is the coherency parameter ($0\le \kappa \le 1$), representing cross-node consistency communication overhead.

The Peak Concurrency Ceiling ($N^{\ast }$)

When coherency overhead is non-zero ($\kappa >0$), adding instances beyond a critical threshold causes throughput to decline (retrograde scaling). The maximum productive scale $N^{\ast }$ is:

Any nodes added beyond $N^{\ast }$ actively degrade cluster performance.

4. Production TypeScript Simulation: USL Model & Open-Loop Benchmark Engine

Below is a simulator that fits USL parameters and demonstrates how open-loop testing captures the true p99 latency during server pauses:

export interface BenchmarkResult {
  concurrency: number;
  measuredThroughput: number;
  meanLatencyMs: number;
  p99LatencyMs: number;
}

export class UniversalScalabilityModel {
  constructor(
    public readonly gamma: number, // Single-worker capacity (e.g. 1000 QPS)
    public readonly sigma: number, // Contention parameter (e.g. 0.05 = 5% serial)
    public readonly kappa: number  // Coherency parameter (e.g. 0.002 = pairwise sync)
  ) {}

  /**
   * Computes expected throughput X(N) using Gunther's USL formula
   */
  public predictThroughput(nodes: number): number {
    const numerator = this.gamma * nodes;
    const denominator = 1 + this.sigma * (nodes - 1) + this.kappa * nodes * (nodes - 1);
    return numerator / denominator;
  }

  /**
   * Calculates the optimal node count N* beyond which retrograde scaling begins
   */
  public calculatePeakScale(): number {
    if (this.kappa <= 0) return Infinity; // Monotonically increases to asymptote
    return Math.sqrt((1 - this.sigma) / this.kappa);
  }
}

export class OpenVsClosedLoopTester {
  /**
   * Simulates how Closed-Loop vs. Open-Loop testing measures p99 during a 3-second server stall
   */
  public static compareLatencyMeasurement(
    targetQPS: number,
    stallDurationSec: number,
    totalDurationSec: number
  ): { closedLoopP99Ms: number; openLoopP99Ms: number } {
    const normalLatencyMs = 10;
    const stallTimeMs = stallDurationSec * 1000;

    // 1. Closed-loop: When server stalls, client stops sending!
    // Out of totalDurationSec * targetQPS, only 1 request experiences the stall per worker
    const closedLoopLatencies: number[] = [];
    const openLoopLatencies: number[] = [];

    const totalRequests = targetQPS * totalDurationSec;
    const stallStart = totalDurationSec / 2;
    const stallEnd = stallStart + stallDurationSec;

    for (let i = 0; i < totalRequests; i++) {
      const scheduledArrivalSec = i / targetQPS;

      // Open-Loop: Requests arrive on schedule regardless of server state
      if (scheduledArrivalSec >= stallStart && scheduledArrivalSec < stallEnd) {
        // Must wait until stall clears + normal latency
        const waitTimeMs = (stallEnd - scheduledArrivalSec) * 1000 + normalLatencyMs;
        openLoopLatencies.push(waitTimeMs);
      } else {
        openLoopLatencies.push(normalLatencyMs);
      }

      // Closed-Loop: Stalls suppress request generation
      if (scheduledArrivalSec >= stallStart && scheduledArrivalSec < stallEnd) {
        if (Math.random() < 0.01) { // Only tiny fraction gets sent
          closedLoopLatencies.push(stallTimeMs);
        }
      } else {
        closedLoopLatencies.push(normalLatencyMs);
      }
    }

    closedLoopLatencies.sort((a, b) => a - b);
    openLoopLatencies.sort((a, b) => a - b);

    const closedLoopP99 = closedLoopLatencies[Math.floor(closedLoopLatencies.length * 0.99)];
    const openLoopP99 = openLoopLatencies[Math.floor(openLoopLatencies.length * 0.99)];

    return {
      closedLoopP99Ms: closedLoopP99,
      openLoopP99Ms: openLoopP99,
    };
  }
}

function runDemo(): void {
  console.log("=== Universal Scalability Model (USL) & Saturation Curve ===");
  const usl = new UniversalScalabilityModel(1000, 0.02, 0.0005);
  console.log("USL Model: gamma=1000 QPS, sigma=0.02 (contention), kappa=0.0005 (coherence)");
  for (const n of [1, 4, 16, 32, 64]) {
    console.log(`  Nodes ${n} -> Predicted Throughput: ${usl.predictThroughput(n).toFixed(0)} QPS`);
  }
}

runDemo();

5. Vertical Scaling (Scale-Up) vs. Horizontal Scaling (Scale-Out)

Before building a multi-node cluster, evaluate the Scale-Up vs. Scale-Out Decision Matrix:

The Pragmatic Rule of Thumb

1. Optimize first: Profile the application with perf and eliminate unnecessary allocations, database queries, and locks.
2. Scale up second: Move to larger instances (e.g., 8 core $\to$ 32 core $\to$ 64 core). A single well-tuned 32-core node can handle up to $20,000\text{ QPS}$ for typical JSON APIs.
3. Scale out third: Transition to horizontal scaling when:
   • High Availability (HA) across physical failure domains / Availability Zones is mandatory.
   • The workload exceeds the compute, memory, or network bounds of the largest cost-effective single machine.

6. Open-Source Tooling & Industry Best Practices

A. wrk2 (Constant-Throughput Open-Loop Benchmarker)

Unlike wrk, wrk2 takes a -R parameter specifying the exact target throughput:

# Generate 10,000 requests/sec across 4 threads and 100 connections for 3 minutes
wrk2 -t4 -c100 -d180s -R10000 --latency http://api.internal.net/v1/checkout

wrk2 outputs an accurate High Dynamic Range (HDR) histogram showing true tail percentiles ($99\%$, $99.9\%$, $99.99\%$) without Coordinated Omission.

B. Linux perf & FlameGraphs

Profile running production processes without instrumentation overhead:

# Capture CPU stack traces across all cores at 99 Hz for 30 seconds
perf record -F 99 -a -g -- sleep 30

# Generate interactive SVG FlameGraph
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu-profile.svg

7. Failure Modes & Operational War Stories

War Story: The Database Cross-Talk Meltdown

• Incident: A company running a sharded database experienced high latency. Management ordered the cluster scaled from $8$ nodes to $32$ nodes to “add capacity.”
• Result: Instead of improving, throughput collapsed by $60\%$ and p99 latency skyrocketed from $25\text{ ms}$ to $800\text{ ms}$.
• Root Cause: The database utilized cross-shard two-phase commit (2PC) and distributed metadata broadcasting. In USL terms, $\kappa =0.008$. At $N=8$, the coherency penalty was manageable. At $N=32$, pairwise coherence messages scaled as $32\times 31=992$ communication channels. The cluster had crossed $N^{\ast }=11$, entering the retrograde collapse zone where nodes spent all their time exchanging synchronization messages instead of executing queries.
• Mitigation: Scaled back down to 10 larger nodes, redesigned the partition key to eliminate cross-shard transactions, reducing $\kappa \approx 0.0001$.

8. Operational Decision Guide

9. Hands-on Operational Drills

1. Calculate USL Peak Node Capacity: Given $\gamma =500\text{ QPS}$, $\sigma =0.02$, and $\kappa =0.003$: $$N^{\ast }=\sqrt{\frac{1-0.02}{0.003}}=\sqrt{\frac{0.98}{0.003}}=\sqrt{326.67}\approx 18\text{ nodes}$$ Calculate maximum throughput $X(18)$: $$X(18)=\frac{500\times 18}{1+0.02(17)+0.003(18)(17)}=\frac{9,000}{1+0.34+0.918}=\frac{9,000}{2.258}\approx 3,985\text{ QPS}$$ Verify that at $N=30$, throughput declines to $3,580\text{ QPS}$ (retrograde scaling).
2. Execute Coordinated Omission Drill: Run OpenVsClosedLoopTester.compareLatencyMeasurement(2000, 3, 30) and verify that closed-loop testing under-reports p99 latency by over $90\%$.

10. Design Alternatives & Scaling Decision Matrix

11. Authoritative References & Version Context

• Neil J. Gunther (2007), “A General Theory of Computational Scalability Based on Rational Functions”, Performance Evaluation / Computing Research Repository — arxiv.org/abs/0808.3024
• Gil Tene (2015), “Understanding Latency and the Coordinated Omission Problem”, Azul Systems Technical Presentation — giltene.github.io/CoordinatedOmission/
• Gene M. Amdahl (1967), “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities”, AFIPS Conference Proceedings — doi.org/10.1145/1465482.1465560
• Version Context: Load generation methodology targets open-loop load engines (wrk2, k6, Locust) on modern Linux 6.x multi-core server environments.