Reverse Proxies, SSL/TLS Termination Architectures & API Gateways

When a user visits a modern application, their device establishes a secure cryptographic session, negotiates protocol versions, and transmits requests that may require authentication, quota validation, and path routing. Exposing application servers directly to this internet traffic is an architectural disaster: application runtimes (Node.js, Python, Java, Go) are optimized for domain business transactions, not for mitigating Slowloris DDoS attacks, buffering low-bandwidth cellular connections, or managing thousands of TLS session handshakes.

This lesson explores the edge tier: Reverse Proxies, SSL/TLS Termination Topologies, and API Gateways.

1. Problem and Motivation

Building high-throughput edge systems requires addressing several fundamental operational challenges:

1. The Slow-Client Worker Exhaustion Problem: A mobile client on a poor 3G connection uploads a $2\text{ MB}$ photo at $20\text{ KB/s}$. If an application server (e.g., Gunicorn or Spring Boot) connects directly to this client, its execution thread is locked waiting for network socket reads for $100\text{ seconds}$. A fleet of $500$ worker threads collapses under just $500$ concurrent slow uploads.
2. Cryptographic CPU Overhead: Modern TLS 1.3 handshakes utilize Elliptic Curve Diffie-Hellman (ECDHE) key exchange and AES-256-GCM symmetric ciphers. Performing asymmetric handshakes on every backend microservice consumes massive CPU cycles and forces every service to manage private keys and certificate lifecycle rotations.
3. Cross-Cutting Concerns Sprawl: If every microservice independently implements rate limiting, JWT token parsing, CORS header validation, and IP blocking, business logic becomes polluted with infrastructure boilerplate.
4. Edge Protocol Impedance: Internet clients connect over HTTP/1.1, HTTP/2, and HTTP/3 (QUIC) over public networks, whereas internal microservices communicate most efficiently over high-performance binary gRPC or multiplexed HTTP/2 connection pools.

2. Mental Model & Architectural Pipeline

SYSTEM ARCHITECTURE BLUEPRINTNETWORKING

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Arch A: Edge Termination & Client

Establishes the ingress and structural boundary across Arch A: Edge Termination & Client: Max Performance • Plaintext Internal.

Architectural Invariant: Max Performance • Plaintext Internal.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. Arch B: TLS Re-Encryption & Client

Coordinates core data-plane execution and state transitions through Arch B: TLS Re-Encryption & Client: Zero-Trust Internal • Double TLS; TLS 1.3; End-to-End Cryptographic Privacy.

Architectural Invariant: Maintains deterministic networking invariants across Arch B: TLS Re-Encryption & Client.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. API GATEWAY MODULAR FILTER CHAIN (Kong / Envoy / APISIX) & Rate Limiting

Enforces durability, state convergence, and output verification across API GATEWAY MODULAR FILTER CHAIN (Kong / Envoy / APISIX) & Rate Limiting: Incoming HTTP; Token Bucket (Redis) IP / API-Key Quotas 429 Too Many Req; JWT Verification (EdDSA) OAuth2 / API Key lookup 401/403 Unauthorized.

Architectural Invariant: gRPC / REST API Clean Headers mTLS / VPC Direct.

[ Internet Client ]
        │ HTTPS (TLS 1.3)
        ▼
[ Reverse Proxy / Edge LB ] ──> Terminates TLS; buffers slow socket I/O; pools upstream conns
        │ HTTP / Cleartext or mTLS
        ▼
[ API Gateway Filter Pipeline ]
  ├─ 1. Rate Limiting Filter (Token Bucket via Redis)
  ├─ 2. Auth Filter (EdDSA / RSA JWT signature verification)
  ├─ 3. Header Mutation & Trace Injection (W3C traceparent)
  └─ 4. Dynamic Router & Circuit Breaker (Canary traffic splitting)
        │
        ▼
[ Internal Microservices Fleet ]

3. Core Terminology

• Reverse Proxy: An intermediate server that sits in front of backend web servers, intercepting requests and returning resources as if it were the origin server.
• TLS Termination: Decrypting encrypted HTTPS client traffic at the edge proxy and forwarding the decrypted traffic (or re-encrypted mTLS) to backend servers.
• TLS Passthrough: Forwarding encrypted raw TCP packets directly to backend servers without decrypting at the intermediate proxy, routing solely based on the unencrypted SNI header.
• mTLS (Mutual TLS): Bidirectional cryptographic authentication where both client and server present and verify X.509 digital certificates.
• Connection Pooling: Pre-establishing and reusing a pool of persistent TCP/HTTP connections from the proxy to backend services, avoiding TCP three-way handshake and slow-start penalties for every request.
• Request Buffering: Storing incoming request headers and bodies in proxy memory or temporary disk before forwarding to backend servers, shielding backends from slow-upload socket stalls.

4. Deep Dive: SSL/TLS Termination Architectures

Architecture A: Edge TLS Termination (Offloading)

• Mechanics: The edge proxy terminates TLS. Traffic traverses the private data center or VPC network as unencrypted HTTP.
• Pros: Lowest backend CPU consumption; simplified certificate management (certificates exist only on edge proxies); backend servers can run lightweight HTTP runtimes.
• Cons: Traffic is unencrypted on internal networks. Violates strict zero-trust compliance standards (e.g., PCI-DSS, HIPAA) if internal network segments are multi-tenant or untrusted.

Architecture B: TLS Termination with Internal Re-Encryption (mTLS)

• Mechanics: Edge proxy terminates client HTTPS, inspects and mutates headers/payloads, and establishes a brand-new internal mutual TLS (mTLS) session to the target service pod using internal private certificates (e.g., issued via HashiCorp Vault or cert-manager).
• Pros: Complete end-to-end cryptographic encryption across all network hops; full Layer 7 visibility at the gateway; strict zero-trust workload identity.
• Cons: Double cryptographic processing overhead; higher edge proxy CPU and latency overhead.

Architecture C: TLS Passthrough

• Mechanics: The edge load balancer operates at Layer 4. It inspects the initial TLS ClientHello packet’s Server Name Indication (SNI) extension to identify the requested hostname (e.g., api.example.com), then forwards the raw encrypted TCP stream directly to the backend pod without decrypting.
• Pros: Complete end-to-end privacy (the load balancer cannot read sensitive client payloads); edge proxy cannot be compromised to steal private keys.
• Cons: Zero Layer 7 visibility at the proxy (no path-based routing, no edge rate limiting, no header transformation, no WAF inspection). Every backend container must independently maintain TLS certificates.

5. Mathematical Foundations & Formal Analysis

5.1 Token Bucket Algorithm for Edge Rate Limiting

The Token Bucket algorithm regulates incoming request bursts. The bucket has capacity $C$ tokens and is replenished at a constant rate of $r$ tokens per second.

Let $T_{\text{last}}$ be the timestamp of the last request, and $T_{\text{now}}$ be the current arrival timestamp. The updated token balance $B(T_{\text{now}})$ is:

If $B(T_{\text{now}})\ge 1$:

• Request is allowed.
• $B(T_{\text{now}})\leftarrow B(T_{\text{now}})-1$.
• $T_{\text{last}}\leftarrow T_{\text{now}}$.

If $B(T_{\text{now}})<1$:

• Request is rejected immediately with HTTP 429 Too Many Requests.
• The Retry-After header is computed as: $$\Delta t_{\text{retry}}=\frac{1-B(T_{\text{now}})}{r}$$

5.2 TLS 1.3 vs. TLS 1.2 Handshake Latency

In TLS 1.2, full session establishment requires 2 Round Trips (2 RTT) before application data can be transmitted: $${\text{Latency}}_{\text{TLS 1.2}}=1\text{ RTT (TCP)}+2\text{ RTT (TLS)}=3\text{ RTT}$$

In TLS 1.3, key exchange (ECDHE) parameters are transmitted directly inside the ClientHello, reducing the handshake to 1 RTT: $${\text{Latency}}_{\text{TLS 1.3}}=1\text{ RTT (TCP)}+1\text{ RTT (TLS)}=2\text{ RTT}$$

With TLS 1.3 Zero Round-Trip Resumption (0-RTT), returning clients can send encrypted application data in the very first flight of packets: $${\text{Latency}}_{\text{TLS 1.3 0-RTT}}=1\text{ RTT}$$

6. Production TypeScript Simulation: API Gateway Filter Chain

Below is an enterprise API Gateway filter pipeline featuring Token Bucket Rate Limiting, JWT Identity Extraction, and Header Sanitization:

export interface GatewayRequest {
  ip: string;
  path: string;
  headers: Record<string, string>;
  body: string;
}

export interface GatewayResponse {
  statusCode: number;
  headers: Record<string, string>;
  body: string;
}

export interface GatewayContext {
  userId?: string;
  tenantId?: string;
  requestId: string;
}

export type GatewayFilter = (
  req: GatewayRequest,
  ctx: GatewayContext,
  next: () => Promise<GatewayResponse>
) => Promise<GatewayResponse>;

export class TokenBucketRateLimiter {
  private tokens: number;
  private lastRefillMs: number;

  constructor(
    private readonly capacity: number,
    private readonly refillRatePerSec: number
  ) {
    this.tokens = capacity;
    this.lastRefillMs = Date.now();
  }

  public tryConsume(): boolean {
    const now = Date.now();
    const elapsedSec = (now - this.lastRefillMs) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsedSec * this.refillRatePerSec);
    this.lastRefillMs = now;

    if (this.tokens >= 1.0) {
      this.tokens -= 1.0;
      return true;
    }
    return false;
  }
}

export class ApiGateway {
  private filters: GatewayFilter[] = [];
  private rateLimiters = new Map<string, TokenBucketRateLimiter>();

  public use(filter: GatewayFilter): void {
    this.filters.push(filter);
  }

  public getRateLimiter(key: string): TokenBucketRateLimiter {
    if (!this.rateLimiters.has(key)) {
      this.rateLimiters.set(key, new TokenBucketRateLimiter(10, 5)); // 10 burst, 5 req/sec
    }
    return this.rateLimiters.get(key)!;
  }

  public async execute(req: GatewayRequest): Promise<GatewayResponse> {
    const ctx: GatewayContext = {
      requestId: `req-${Math.random().toString(36).substring(2, 9)}`,
    };

    let index = 0;
    const next = async (): Promise<GatewayResponse> => {
      if (index < this.filters.length) {
        const filter = this.filters[index++];
        return filter(req, ctx, next);
      }
      // Target Upstream Service Execution (Mock)
      return {
        statusCode: 200,
        headers: {
          "Content-Type": "application/json",
          "X-Request-Id": ctx.requestId,
          "x-latency-ms": "1.2",
        },
        body: JSON.stringify({ message: "Success", user: ctx.userId }),
      };
    };

    return next();
  }
}

// Instantiate Gateway with Production Filter Pipeline
export function createProductionGateway(): ApiGateway {
  const gw = new ApiGateway();

  // 1. Rate Limiting Filter
  gw.use(async (req, ctx, next) => {
    const clientKey = req.headers["x-api-key"] || req.ip;
    const limiter = gw.getRateLimiter(clientKey);
    if (!limiter.tryConsume()) {
      return {
        statusCode: 429,
        headers: { "Retry-After": "1" },
        body: JSON.stringify({ error: "Too Many Requests" }),
      };
    }
    return next();
  });

  // 2. Authentication Filter
  gw.use(async (req, ctx, next) => {
    const authHeader = req.headers["authorization"];
    if (!authHeader || !authHeader.startsWith("Bearer ")) {
      return {
        statusCode: 401,
        headers: { "WWW-Authenticate": "Bearer" },
        body: JSON.stringify({ error: "Missing or invalid bearer token" }),
      };
    }
    const token = authHeader.substring(7);
    // Simplified JWT subject extraction
    if (token === "invalid-token") {
      return {
        statusCode: 403,
        headers: {},
        body: JSON.stringify({ error: "Forbidden: Signature invalid" }),
      };
    }
    ctx.userId = "usr-4299";
    ctx.tenantId = "tenant-enterprise";
    return next();
  });

  // 3. Header Sanitization & Context Injection Filter
  gw.use(async (req, ctx, next) => {
    // Strip untrusted incoming internal headers
    delete req.headers["x-internal-user-id"];
    // Inject verified identity down to upstream services
    req.headers["x-user-id"] = ctx.userId!;
    req.headers["x-tenant-id"] = ctx.tenantId!;
    req.headers["x-request-id"] = ctx.requestId;

    return next();
  });

  return gw;
}

async function runDemo(): Promise<void> {
  console.log("=== Reverse Proxies & Production API Gateway Simulation ===");
  const gateway = createProductionGateway();
  const req1: GatewayRequest = {
    ip: "203.0.113.42",
    path: "/api/v1/orders",
    headers: {
      "x-api-key": "user-pro-100",
      authorization: "Bearer valid-jwt-token",
    },
    body: "",
  };
  const res1 = await gateway.execute(req1);
  console.log(
    `Handled API Request: status=${res1.statusCode}, latency=${res1.headers["x-latency-ms"]}ms, body=${res1.body}`
  );
}

runDemo();

7. Open-Source Implementation Analysis

A. NGINX: Connection Pooling & Request Buffering

In NGINX, reverse proxy behavior is governed by the ngx_http_proxy_module:

upstream backend_cluster {
    server 10.0.1.10:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    keepalive 64; # Maintains 64 idle connections per worker to upstream
}

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate     /etc/ssl/certs/api.crt;
    ssl_certificate_key /etc/ssl/private/api.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    proxy_buffering on;
    proxy_buffer_size 8k;
    proxy_buffers 64 8k;

    location / {
        proxy_pass http://backend_cluster;
        proxy_http_version 1.1;
        proxy_set_header Connection ""; # Clear Connection: close for keepalive
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

B. Kong Gateway: Lua Plugin Execution Pipeline

Kong runs atop OpenResty (NGINX + LuaJIT). It breaks request processing into discrete phases:

• certificate: Dynamic SNI certificate resolution from database without config reload.
• rewrite: Path mutations and header rewriting.
• access: Executes rate limiting, OAuth2/JWT verification, and IP ACLs before upstream proxying.
• header_filter & body_filter: Response inspection, compression, and data transformation.

8. Failure Modes & Operational War Stories

War Story: The HTTP Request Smuggling Attack (CL.TE Desync)

• Vulnerability: An e-commerce platform ran NGINX as the edge reverse proxy and a legacy Java application server on the backend over persistent HTTP/1.1 connections.
• Root Cause: An attacker submitted an HTTP request containing both:

  POST / HTTP/1.1
  Host: vulnerable.com
  Content-Length: 13
  Transfer-Encoding: chunked
  
  0
  
  SMUGGLED

  NGINX prioritized Content-Length: 13 and forwarded all bytes to the backend. The Java backend prioritized Transfer-Encoding: chunked, terminated the chunk at 0, and interpreted the leftover SMUGGLED string as the starting prefix of the next innocent customer’s incoming request!
• Mitigation:
  1. Configured NGINX with proxy_http_version 1.1; and disabled ambiguity handling.
  2. Mandated HTTP/2 or gRPC between edge proxies and backend servers, eliminating chunked text ambiguity completely.

9. Operational Decision Guide

10. Hands-on Operational Drills

1. Verify Token Bucket Quota Exhaustion: Execute 20 concurrent requests against the TypeScript simulation. Verify that requests 1–10 succeed with 200, while requests 11–20 are rejected with HTTP 429 and valid Retry-After headers.
2. Inspect Upstream Keep-Alive Re-use: Use tcpdump -nn -i lo port 8080 while sending 100 HTTP requests through NGINX to observe zero SYN/FIN packets after the initial connection handshake.

export async function verifyTokenBucketBurst(): Promise<void> {
  const gateway = createProductionGateway(); // Capacity C = 10, refill r = 5 tokens/sec
  const statuses: number[] = [];

  for (let i = 1; i <= 20; i++) {
    const res = await gateway.execute({
      ip: '203.0.113.42',
      path: '/api/v1/orders',
      headers: {
        'x-api-key': 'tenant-burst-test',
        authorization: 'Bearer valid-jwt-token',
      },
      body: '',
    });
    statuses.push(res.statusCode);
  }

  console.log(`Requests 1..10 statuses: ${statuses.slice(0, 10).join(', ')}`);   // 200 x 10
  console.log(`Requests 11..20 statuses: ${statuses.slice(10, 20).join(', ')}`); // 429 x 10
}

11. Design Alternatives & Gateway Trade-Offs

12. Authoritative References & Version Context

• Eric Rescorla (2018), “The Transport Layer Security (TLS) Protocol Version 1.3”, IETF RFC 8446 — datatracker.ietf.org/doc/html/rfc8446
• Mark Nottingham and Roy Fielding (2022), “HTTP Semantics”, IETF RFC 9110 — datatracker.ietf.org/doc/html/rfc9110
• Envoy Project Documentation (v1.30+), Architecture Overview: Threading Model and Dynamic xDS Configuration — envoyproxy.io/docs/envoy/latest/intro/arch_overview/arch_overview
• Version Context: Specifications align with TLS 1.3 (RFC 8446), HTTP/2 (RFC 9113), HTTP/3 (RFC 9114), Envoy 1.30+, and NGINX 1.26 LTS.