Traffic Ingress Architecture: L4 Transport vs. L7 Application Load Balancing

Every distributed application—whether handling payment webhooks, video streaming chunks, or real-time chat websockets—begins at the traffic ingress layer. Long before a request invokes business logic, executes a database transaction, or checks a distributed cache, it must cross the boundary between the untrusted public internet and internal cluster networks.

A fundamental design failure in engineering occurs when developers treat load balancers as magical black boxes that simply “spread traffic evenly.” In production systems, how ingress traffic is routed, terminated, and scheduled determines tail latencies, resilience against cascading brownouts, and operational safety during rolling upgrades.

1. Problem and Motivation

Consider an e-commerce platform during a flash sale receiving $250,000$ incoming HTTP requests per second across $100$ application servers.

If incoming traffic is routed naively:

1. Uneven Server Utilization: Simple round-robin routing assumes every request requires identical compute resources. If Server A receives five complex catalog search queries while Server B receives five lightweight static health checks, Server A saturates its CPU and queue depth while Server B sits idle.
2. The Fast-Failure Stampede: If Server C begins throwing immediate database connection timeouts ($1\text{ ms}$ error latency), its connections close almost instantly. A “Least Connections” algorithm observes near-zero connections on Server C and mistakenly diverts massive traffic to the broken node, accelerating systemic failure.
3. The Ingress Bottleneck: Terminating TLS handshakes and parsing HTTP headers on a single monolithic reverse proxy requires substantial CPU cycles. When network traffic hits $40\text{ Gbps}$, standard Linux kernel socket layers experience packet drops from context switches and CPU software interrupt (softirq) saturation.
4. Deploy Interruptions: When deploying a new container revision, instantly killing old pods drops thousands of active, in-flight HTTP connections, resulting in client-visible 502 Bad Gateway and 504 Gateway Timeout errors.

To build predictable systems, engineers must architect a multi-tiered ingress topology separating Layer 4 transport dispatching from Layer 7 application routing.

2. Mental Model & Ingress Architecture

Traffic ingress is structured as a pipeline of progressively specialized filters:

SYSTEM ARCHITECTURE BLUEPRINTNETWORKING

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Client Handshake & BGP Anycast ECMP

Client establishes HTTPS connection. Router performs Equal-Cost Multi-Path (ECMP) 5-tuple hashing (SrcIP, DstIP, SrcPort, DstPort, Protocol) to spread flows across balancers.

Architectural Invariant: Flow Affinity: ECMP ensures all packets in the same TCP connection consistently hit the exact same load balancer instance.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. Layer 7 Proxy TLS Decryption & Path Routing

Envoy terminates TLS session, parses HTTP path and headers, and executes Power of Two Random Choices (P2C) algorithm to evaluate backend health and latency.

Architectural Invariant: P2C Spreading: Picking the least-loaded of two random candidates provides optimal O(log log n) load distribution with zero global coordination.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Multiplexed Dispatch to Healthy Service B

Envoy selects Service Instance B (least active connections = 12). Request streams over a pre-warmed persistent HTTP/2 connection pool.

Architectural Invariant: Connection Pooling: Reusing established backend TCP/TLS connections eliminates three-way handshake and TLS negotiation latencies.

[ Client (Browser/App) ]
           │
           ▼
[ BGP Anycast Routing ] ──> Routes client IP to closest Edge Point of Presence (PoP)
           │
           ▼
[ Equal-Cost Multi-Path (ECMP) ] ──> Top-of-Rack Switch hashes 5-tuple to L4 Balancers
           │
           ▼
[ Layer 4 Balancers (IPVS/Maglev) ] ──> Direct Server Return (DSR); dispatches TCP flows
           │
           ▼
[ Layer 7 Proxies (Envoy/HAProxy) ] ──> Terminates TLS; inspects HTTP path/headers; routes
           │
           ▼
[ Backend Application Fleets ] ──> Service containers / pods executing application logic

The Fundamental Separation: L4 vs. L7

3. Core Terminology & Architectural Concepts

• VIP (Virtual IP): A single public IP address advertised to the internet via BGP Anycast. Multiple physical load balancer instances announce the same VIP.
• 5-Tuple Flow: The unique signature of a TCP/UDP connection: (Source IP, Destination IP, Source Port, Destination Port, Protocol).
• DSR (Direct Server Return): An L4 acceleration technique where incoming request packets pass through the load balancer, but outgoing response packets bypass the load balancer and return directly from backend servers to clients through the router.
• Connection Draining (Deregistration Delay): Allowing active, in-flight transactions to complete over existing TCP connections while blocking new connection dispatches to a shutting-down backend.
• Outlier Detection (Passive Health Checking): Dynamically detecting backends returning consecutive 5xx errors or excessive latency during live production traffic, ejecting them temporarily from the routing pool.
• Power of Two Random Choices (P2C): An algorithm that selects two random backend servers and routes traffic to the one with lower active load.

4. Internal Mechanics & Routing Protocols

4.1 Layer 4 Direct Server Return (DSR) Mechanics

In standard proxying or NAT (Network Address Translation), all request and response bytes traverse the load balancer:

Because HTTP request bodies (e.g., GET /index.html) are typically less than $1\text{ KB}$ while responses (JSON, images, video segments) are often $50\text{ KB}-10\text{ MB}$, return bandwidth saturates the load balancer’s network interface card (NIC).

DSR resolves this asymmetry:

1. Client sends a packet with SrcIP = ClientIP, DstIP = VIP.
2. The L4 load balancer does not change the IP addresses. Instead, it encapsulates the packet in Generic UDP Encapsulation (GUE) or rewrites the Layer 2 Destination MAC address to point to Backend Server A.
3. Backend Server A has the VIP configured on a virtual non-arping loopback interface (lo:0).
4. Backend Server A decapsulates the packet, processes the request, and produces a response with SrcIP = VIP, DstIP = ClientIP.
5. Backend Server A sends the response directly out its local network gateway to the client. The load balancer never sees the response bytes.

4.2 Layer 7 Balancing Algorithms

When requests reach L7 proxies (e.g., Envoy or HAProxy), the connection is terminated and content is parsed. Balancers employ several algorithms:

A. Weighted Round Robin (WRR)

Requests are dispatched cyclically according to declared static integer weights: $$S=\{(b_1,w_1),(b_2,w_2),\dots ,(b_n,w_n)\}$$ Smooth round-robin interleaves servers to avoid burst-loading high-weight servers.

B. Least Connections

Dispatches the new request to the server with the lowest number of active concurrent TCP streams or HTTP/2 streams: $$b^{\ast }=\arg \underset{b\in B}{\min }\left(\frac{\text{ActiveConns}(b)}{\text{Weight}(b)}\right)$$

C. Power of Two Random Choices (P2C) with Peak EWMA

Rather than scanning all $N$ servers ($O(N)$ overhead) or blindly choosing the least-connected (which causes herd behavior), P2C chooses two candidates uniformly at random: $$\text{Candidates}=\{b_i,b_j\}(i,j\sim \text{Uniform}(1,N),i\ne j)$$ It scores both candidates using exponentially weighted moving average (EWMA) latency and active connections: $$\text{Score}(b)=\text{ActiveConns}(b)\times {\text{EWMA}}_{\text{Latency}}(b)$$ The candidate with $\min (\text{Score}(b_i),\text{Score}(b_j))$ wins.

5. Mathematical Foundations & Formal Analysis

5.1 Mitzenmacher’s Theorem: The Power of Two Choices

Suppose $N$ requests are assigned to $N$ backend servers.

• Under pure random selection, the maximum number of requests assigned to any server with high probability is: $${\text{Max Load}}_{\text{Random}}=\frac{\log N}{\log \log N}\cdot (1+o(1))$$
• Under Power of Two Random Choices ($d=2$), where each request samples two nodes and picks the one with lower load, the maximum load drops exponentially to: $${\text{Max Load}}_{d=2}=\frac{\log \log N}{\log 2}+O(1)$$

For $N=1,000,000$ servers:

• Pure random allocation yields a peak queue depth of approximately $\approx 7$.
• Choosing the best of two reduces the peak queue depth to $\approx 3$.

5.2 Outlier Detection Consecutive Error Probability

Let $p$ be the baseline random network error rate of healthy backends ($p\approx 0.001$). Let $k$ be the consecutive error ejection threshold (e.g., $k=5$). The false-positive probability of healthy server $b$ being falsely ejected after $m$ requests is bounded by: $$P(\text{False Ejection})\approx 1-(1-p^k{)}^m$$ For $p=10^{-3}$ and $k=5$, $p^5=10^{-15}$. Over $m=10^7$ requests, $P(\text{False Ejection})\approx 10^{-8}$ (virtually zero). However, a failing node with an error rate $p=0.5$ will trigger $0.5^5=0.03125$, suffering rapid ejection within $\approx 32$ requests.

6. Concrete Architecture & Component Topology

               Public Internet
                     │
         [ VIP: 198.51.100.1 announced via BGP Anycast ]
                     │
              Top-of-Rack Switch
          (ECMP Hashing over 5-tuple)
            ┌────────┴────────┐
            ▼                 ▼
     [ L4 Katran/IPVS ] [ L4 Katran/IPVS ]
            │ (DSR / Encapsulated Ingress)
            └────────┬────────┘
                     ▼
         [ L7 Envoy Gateway Tier ]
     (Terminates TLS, Rate Limits, P2C)
       ┌───────────┼───────────┐
       ▼           ▼           ▼
   Backend-1   Backend-2   Backend-3 (Drain)

7. Production TypeScript Simulation: P2C Load Balancer

Here is an executable implementation of a Layer 7 load balancer featuring Power of Two Random Choices (P2C), Peak EWMA latency tracking, Outlier Detection, and Graceful Connection Draining:

export interface BackendNode {
  id: string;
  activeRequests: number;
  ewmaLatencyMs: number;
  isDraining: boolean;
  consecutive5xx: number;
  ejectedUntilMs: number;
}

export class P2CLoadBalancer {
  private backends: BackendNode[] = [];
  private readonly alpha = 0.2; // EWMA smoothing factor
  private readonly ejectionThreshold = 5;
  private readonly ejectionDurationMs = 30000;

  constructor(backendIds: string[]) {
    this.backends = backendIds.map((id) => ({
      id,
      activeRequests: 0,
      ewmaLatencyMs: 10, // initial optimistic seed
      isDraining: false,
      consecutive5xx: 0,
      ejectedUntilMs: 0,
    }));
  }

  public getAvailableBackends(): BackendNode[] {
    const now = Date.now();
    return this.backends.filter(
      (b) => !b.isDraining && b.ejectedUntilMs <= now
    );
  }

  public selectNode(): BackendNode {
    const pool = this.getAvailableBackends();
    if (pool.length === 0) {
      throw new Error("503 Service Unavailable: All backend nodes ejected or draining");
    }
    if (pool.length === 1) {
      return pool[0];
    }

    // Power of Two Random Choices (P2C)
    const idx1 = Math.floor(Math.random() * pool.length);
    let idx2 = Math.floor(Math.random() * (pool.length - 1));
    if (idx2 >= idx1) idx2++;

    const nodeA = pool[idx1];
    const nodeB = pool[idx2];

    // Compute composite load score: Active Connections * Latency
    const scoreA = (nodeA.activeRequests + 1) * nodeA.ewmaLatencyMs;
    const scoreB = (nodeB.activeRequests + 1) * nodeB.ewmaLatencyMs;

    return scoreA <= scoreB ? nodeA : nodeB;
  }

  public async dispatch<T>(fn: (node: BackendNode) => Promise<{ status: number; data: T }>): Promise<T> {
    const node = this.selectNode();
    node.activeRequests++;
    const start = performance.now();

    try {
      const response = await fn(node);
      const elapsed = performance.now() - start;

      // Update Peak EWMA Latency
      node.ewmaLatencyMs = this.alpha * elapsed + (1 - this.alpha) * node.ewmaLatencyMs;

      if (response.status >= 500) {
        node.consecutive5xx++;
        if (node.consecutive5xx >= this.ejectionThreshold) {
          node.ejectedUntilMs = Date.now() + this.ejectionDurationMs;
          node.consecutive5xx = 0;
        }
      } else {
        node.consecutive5xx = 0;
      }

      return response.data;
    } catch (err) {
      node.consecutive5xx++;
      if (node.consecutive5xx >= this.ejectionThreshold) {
        node.ejectedUntilMs = Date.now() + this.ejectionDurationMs;
        node.consecutive5xx = 0;
      }
      throw err;
    } finally {
      node.activeRequests--;
    }
  }

  public startDrain(nodeId: string): void {
    const node = this.backends.find((b) => b.id === nodeId);
    if (node) {
      node.isDraining = true;
    }
  }
}

async function runDemo(): Promise<void> {
  console.log("=== Load Balancing: Power-of-Two-Random-Choices (P2C) ===");
  const p2c = new P2CLoadBalancer(["node-1", "node-2", "node-3", "node-4"]);
  console.log("Initialized P2C Load Balancer with 4 backends");
  for (let i = 0; i < 5; i++) {
    const selected = p2c.selectNode();
    console.log(`Request ${i + 1} routed to: ${selected.id} (active: ${selected.activeRequests})`);
  }
}

runDemo();

8. Real-World Open-Source Implementations

A. HAProxy (High Performance L4/L7)

HAProxy is written in C and runs on an event-driven architecture using epoll(7). Key operational capabilities include:

• Stick Tables: Shared-memory distributed key-value tables tracking client IP request rates, connection rates, and error rates across multi-process instances for edge DDoS mitigation.
• Server State Persistence: Persists health state across configuration reloads using Unix domain sockets (/var/run/haproxy.sock), preventing the “cold reload stampede” where every backend is aggressively re-probed simultaneously.

B. Envoy Proxy (Cloud-Native Service Mesh Ingress)

Envoy uses a single-process, multi-threaded architecture where one thread per hardware core runs an independent event loop. Key mechanisms:

• xDS Control Plane: Ingress routing tables (RDS), listener bindings (LDS), and cluster endpoints (EDS) are updated dynamically via gRPC streaming with zero process reloads.
• Outlier Detection Filter: Continuously calculates the mean and standard deviation of 5xx return codes across the cluster, ejecting statistically abnormal backends with exponential backoff timers.

9. Failure Modes & Production War Stories

War Story 1: The Fast-Failing Node Catastrophe

• Incident: During peak Black Friday traffic, a database node exhausted connections. One API container threw 500 Internal Server Error instantly ($0.4\text{ ms}$ processing time), while healthy nodes took $85\text{ ms}$ executing valid database queries.
• Root Cause: The load balancer was configured with standard Least Connections. Because the broken node closed sockets instantly, its active connection count hovered between $0$ and $1$, while healthy servers held $80$ concurrent connections. The balancer routed $85\%$ of all new requests to the broken server, amplifying failure.
• Mitigation: Migrated to P2C combined with Peak EWMA and configured Outlier Detection: eject any node exceeding 5 consecutive 5xx responses.

War Story 2: Connection Draining Deadlock during Rolling Updates

• Incident: A rolling Kubernetes deploy took 45 minutes instead of 3 minutes.
• Root Cause: The Ingress controller had connection-draining-timeout: 3600 configured, and clients kept persistent HTTP/1.1 Keep-Alive connections open indefinitely sending idle pings.
• Mitigation: Configured server-side keepalive_requests 1000; and sent Connection: close headers to active clients during SIGTERM shutdown.

10. Operational Decision Guide

11. Hands-on Operational Drills

1. Simulate Outlier Ejection: Run the TypeScript simulation with 4 nodes. Inject an intentional error loop on Node 3 and observe the balancer automatically ejecting Node 3 after 5 errors.
2. Benchmark P2C vs Random: Simulate $10,000$ concurrent requests across $20$ nodes with variable latency ($5\text{ ms}$ to $200\text{ ms}$). Plot the standard deviation of queue depth across backends under pure random vs. P2C.

export async function verifyOutlierEjection(): Promise<void> {
  const lb = new P2CLoadBalancer(['node-1', 'node-2', 'node-3', 'node-4']);

  // Simulate node-3 returning immediate HTTP 503 errors
  for (let attempt = 1; attempt <= 40; attempt++) {
    await lb.dispatch(async (node) => {
      if (node.id === 'node-3') {
        return { status: 503, data: 'DB pool exhausted' };
      }
      return { status: 200, data: 'OK' };
    });
  }

  const healthyIds = lb.getAvailableBackends().map((b) => b.id);
  console.log(`Active routing pool after fault injection: [${healthyIds.join(', ')}]`);
  // Output: Active routing pool after fault injection: [node-1, node-2, node-4]
}

12. Design Alternatives & Load Balancing Trade-Offs

13. Authoritative References & Version Context

• Daniel E. Eisenbud et al. (2016), “Maglev: A Fast and Reliable Software Network Load Balancer”, USENIX NSDI ‘16 (Google Maglev architecture) — usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud
• Nikita Shirokov and Ranjeeth Dasari (2018), “Open-sourcing Katran, a scalable network load balancer”, Meta Engineering Blog — engineering.fb.com/2018/05/22/open-source/katran-a-scalable-network-load-balancer/
• HAProxy Technologies (2024), “HAProxy Architecture and Direct Server Return (DSR) Guide” — haproxy.com/documentation/
• Version Context: Protocols reflect BGP RFC 4271, ECMP RFC 2992, and Linux kernel 5.x/6.x XDP/eBPF packet processing baselines.