At the edge of modern distributed infrastructures, reverse proxies and API gateways are the primary entry points for planetary-scale traffic. Every incoming HTTP, gRPC, and WebSocket request traverses these edge layers before reaching internal services. Achieving sub-millisecond routing, TLS termination, load balancing, rate limiting, and authentication across hundreds of thousands of concurrent connections requires specialized operating system primitives, lock-free memory layouts, and asynchronous event loops.
This architectural study deconstructs the foundational systems powering edge routing:
- HAProxy: The gold standard in high-performance Layer 4 and Layer 7 proxying, featuring zero-copy TCP splicing, lock-free stick tables, and socket-passing hitless reloads.
- NGINX & OpenResty: The event-driven master-worker paradigm,
epollnotification loops, and LuaJIT coroutine cosockets. - Kong & Apache APISIX: Next-generation programmable API gateways comparing RDBMS-backed polling and invalidation clusters against distributed etcd v3 MVCC streaming.
1. Context & Architecture Setting
Project Identity, Version & License Baseline
| Project Component | Canonical Repository | Pinned Release | Governance / License | Edition Boundary |
|---|---|---|---|---|
| HAProxy | haproxy/haproxy | v2.8 (LTS) | Willy Tarreau / Community (GPL v2 / LGPL 2.1) | Open source |
| NGINX | nginx/nginx | v1.26 (Stable) | F5 / NGINX Open Source (2-clause BSD) | Open source core vs NGINX Plus |
| Kong Gateway | Kong/kong | v3.6 | Kong Inc (Apache 2.0) | Open source core vs Kong Enterprise |
| Apache APISIX | apache/apisix | v3.8 | Apache Software Foundation (Apache 2.0) | Open source |
Every edge gateway faces a severe concurrency challenge known historically as the C10K (and now C10M) problem:
- Thread-per-connection exhaustion: Early servers (such as Apache HTTPD
preforkor naive thread pools) allocated one operating system thread per connection. At concurrent connections, stack allocations alone ( per thread) consume hundreds of gigabytes of RAM, while Linux kernel O(N) context switching renders CPU caches useless. - Synchronous I/O stalls: If a worker blocks waiting for an upstream database query or backend microservice response, that compute resource cannot serve other incoming connections.
- Dynamic policy application: Modern cloud-native architectures require modifying routing rules, rate limits, and authentication keys hundreds of times per day without dropping in-flight TCP handshakes or causing latency spikes.
+-----------------------------------------------------------------------------+
| EDGE ARCHITECTURE SPECTRUM |
+-----------------------------------------------------------------------------+
| Layer 4 (Transport) | Layer 7 (Application) | Programmable Gateway |
| HAProxy / IPVS | NGINX / Envoy | Kong / Apache APISIX |
| | | |
| - TCP/UDP stream proxy | - HTTP/1.1, HTTP/2, gRPC | - Dynamic Auth (JWT) |
| - Zero-copy splice() | - TLS Termination / SNI | - Distributed Limits |
| - Stick table counters | - Header rewrite / cache | - Lua / Wasm plugins |
| - Kernel socket handoff | - epoll event loops | - etcd / DB-less sync |
+-----------------------------------------------------------------------------+
2. Architectural Mental Model
1. Layer 4 Ingress and HAProxy Core & Linux Kernel: SO_REUSEPORT
Establishes the ingress and structural boundary across Layer 4 Ingress and HAProxy Core & Linux Kernel: SO_REUSEPORT: Kernel 4-tuple hash distributes incoming SYN Zero-lock lockless connection distribution; Single-process multi-threaded event loop; Zero-copy TCP splicing (splice syscall).
2. NGINX Core and OpenResty Architecture & Master Process (Privileged PID 1)
Coordinates core data-plane execution and state transitions through NGINX Core and OpenResty Architecture & Master Process (Privileged PID 1): Reads and validates nginx.conf; opens sockets; Forks workers; orchestrates hot reload via SIGHUP; epoll_wait() loop Non-blocking state LuaJIT VM instance 100k+ keepalive conns.
3. Gateway Control Planes: Kong vs APISIX & Kong Gateway Control Plane
Enforces durability, state convergence, and output verification across Gateway Control Planes: Kong vs APISIX & Kong Gateway Control Plane: Storage: PostgreSQL / Cassandra or DB-less YAML; Invalidation: Cluster events bus over pub/sub; Worker Cache: Local memory LRU cache per node ⚠ Cache warm-up latency and DB polling overhead.
To understand how high-throughput proxies operate, we must contrast the process and concurrency models of HAProxy, NGINX, and OpenResty-based Gateways.
The HAProxy Model: Multi-Threaded Single Process
HAProxy operates as a single master process running multiple worker threads pinned to CPU cores.
- Unified Event Scheduler: Each thread runs an event loop monitoring file descriptors via Linux
epollor BSDkqueue. - Zero-Copy TCP Splicing: For Layer 4 proxying, HAProxy uses
splice(2). Network data moves directly from the incoming socket buffer into the outgoing socket buffer inside kernel memory, never copying payload bytes into user-space buffers. - Stick Tables: An in-memory, lock-free key-value store using Elastic Binary Trees (
ebtree) that tracks client IP connection rates, error counts, and session cookies across threads and peered cluster nodes.
The NGINX Model: Master-Worker Event Loop
NGINX isolates concurrency across independent worker processes rather than threads:
- Master Process: Runs with privileged access, parses configuration files, binds to listening sockets (e.g., ports 80 and 443), and manages worker lifecycles.
- Worker Processes: Unprivileged processes forked from the master. Each worker runs a single-threaded non-blocking event loop. Connections are distributed via Linux
SO_REUSEPORT, allowing the kernel networking stack to load-balance incoming SYN packets across worker listening sockets without lock contention. - Shared Memory Slabs: Workers communicate shared state (rate limits, SSL session caches, upstream health) through POSIX shared memory segments (
shm) protected by lightweight atomic spinlocks.
The OpenResty / LuaJIT Gateway Model
OpenResty embeds the LuaJIT compiler inside NGINX worker processes:
- Cosocket Mechanism: When Lua code inside a plugin performs network I/O (e.g., querying Redis or verifying an OAuth token), OpenResty yields the Lua coroutine. The underlying socket is registered with the NGINX
epollloop. When data arrives, NGINX resumes the coroutine. - Developer Simplicity, Async Speed: Engineers write straightforward procedural code (
local res = http:request(...)) while the underlying engine executes fully non-blocking I/O.
3. Core Mechanisms & Protocol Internals
Master-Worker Event Loops & Socket Inheritance
When NGINX initializes:
- The master process calls
socket(),setsockopt(SO_REUSEPORT),bind(), andlisten(). - Master forks worker processes (typically ).
- Each worker process inherits the file descriptors and executes:
- When a reload signal (
SIGHUP) is sent, the master spawns new workers with the updated configuration. Old workers stop accepting new connections, finish serving active connections within a grace period, and terminate cleanly.
[Client Traffic]
|
v (Kernel 4-tuple hashing)
+-----------------+
| SO_REUSEPORT |
+--------+--------+
|
+------+------+
| |
v v
+--------------+ +--------------+
| Worker 0 | | Worker 1 |
| epoll_wait() | | epoll_wait() |
| LuaJIT VM | | LuaJIT VM |
+-------+------+ +-------+------+
| |
+--------+-------+
|
v
[lua_shared_dict Slab]
(Atomic CAS Rate Limits)
HAProxy Zero-Copy Splicing
In standard user-space proxies, relaying a packet requires two system calls and two memory copies:
read(client_fd, buf, len)copies kernel receive buffer to user memory.write(backend_fd, buf, len)copies user memory to kernel send buffer.
HAProxy avoids this through Linux pipe buffers: No payload bytes cross the kernel-user boundary. The CPU only manipulates buffer pointer descriptors, slashing memory bus traffic and CPU cache misses.
HAProxy Stick Tables & Cluster Synchronization
Stick tables provide stateful distributed intelligence at line rate:
- Elastic Binary Trees (
ebtree): O(log N) lookup and insertion with zero memory fragmentation. - Peering Protocol: HAProxy nodes stream stick table state changes over dedicated TCP connections (
peerssection) using a compact binary protocol, providing real-time global DDoS protection across a cluster.
4. Gateway Control Plane Distribution: Kong vs Apache APISIX
Programmable API gateways differ fundamentally in how they distribute configuration changes to worker data planes:
| Architectural Dimension | Kong Gateway (Traditional) | Apache APISIX |
|---|---|---|
| Configuration Store | PostgreSQL / Cassandra or DB-less YAML | Distributed etcd v3 cluster (MVCC) |
| Worker Synchronization | Polling database + cluster event pub/sub | HTTP/2 long-polling Watch API |
| Propagation Latency | Hundreds of milliseconds to seconds | Under 5 milliseconds |
| Routing Algorithm | Lua-based prefix tree matching | Radix Tree (rax) implemented in C |
| Process Model | OpenResty NGINX wrapper | OpenResty with optimized C libraries |
| Dynamic SSL / Upstreams | Cached in lua_shared_dict via polling | Streamed via etcd watch into shared memory |
Apache APISIX etcd Watch Architecture
Rather than polling a relational database, APISIX workers establish long-lived HTTP/2 streams to etcd:
- An administrator posts a route change to the Admin API:
PUT /apisix/admin/routes/1. - etcd persists the key-value revision in its Raft-backed Bbolt engine.
- etcd immediately pushes the revision delta over the active
Watchconnection to all APISIX worker nodes. - Each worker updates its in-memory Radix tree atomically using pointer swaps. Zero master reloads, zero dropped connections.
5. Mathematical & Performance Analysis
Ingress Queueing & Little’s Law
The concurrency capacity of an edge proxy is governed by Little’s Law: Where:
- = Average number of concurrent open connections inside the proxy.
- = Ingress arrival rate (requests per second).
- = Total connection residence time (including upstream latency and network transfer).
If an edge proxy receives and the mean upstream latency is :
However, if an upstream service degrades and climbs to under a retry storm:
If the proxy worker limit (worker_connections) is configured to , the proxy runs out of file descriptors, rejecting new client connections with HTTP 502 / 504 errors.
Memory Footprint of Concurrent Connections
The RAM requirement of an event-driven reverse proxy is calculated as: Where:
- = Linux kernel socket buffer size (
rmem+wmemtuned). - = Proxy user-space buffer (
proxy_buffer_size). - At concurrent idle keepalive connections:
An event-driven proxy handles 1M idle connections in 8 GB RAM, whereas a thread-per-connection architecture would require .
6. Concrete Failure Modes & Edge Cases
1. The OpenResty Coroutine Blocking Trap
- Failure Mode: An engineer imports a third-party Lua library that uses standard Lua
socketor C-FFI synchronous file I/O instead ofngx.socket.tcp(). - System Impact: The entire single-threaded NGINX worker stops processing events. All concurrent requests on that worker stall until the synchronous I/O unblocks or times out, causing latency spikes that alternate across CPU cores.
- Remediation: Enforce automated linting to ban standard Lua
io.*and blocking sockets. Use OpenResty cosocket-compatible drivers (lua-resty-redis,lua-resty-http).
2. NGINX Master Reload Drop During High Connection Velocity
- Failure Mode: Under heavy traffic (), executing
nginx -s reloadtriggers SIGHUP. - System Impact: Old workers close their listen sockets and enter drain mode. If new workers take several milliseconds to initialize Lua VMs and compile regex routes, the Linux socket listen backlog (
backlogparameter inlisten) overflows, dropping SYN packets and causing TCP connection timeouts. - Remediation: Use
reuseportwith HAProxy hitless reload (-xsocket handoff) or use dynamic control planes like APISIX/Envoy that update routes via memory pointers without process restarts.
3. lua_shared_dict Spinlock Contention
- Failure Mode: Hundreds of concurrent requests on high-core servers () increment a single global rate-limit key in
lua_shared_dict. - System Impact: OpenResty’s shared memory dictionary uses atomic CPU spinlocks. Severe contention across 128 cores causes CPU utilization to spike to 100% in kernel spin-lock loops without processing useful traffic.
- Remediation: Shard shared memory keys across multiple buckets (e.g.,
key .. ":" .. (worker_id % 16)), or use local worker caching with periodic batch synchronization.
7. Production Trade-Offs & Anti-Patterns
Trade-Off Matrix
| Strategy | Advantages | Disadvantages | Optimal Use Case |
|---|---|---|---|
| Pure HAProxy (L4) | Unbeatable throughput, minimal memory, kernel splice() | No payload transformation, limited dynamic routing | TCP/UDP ingress, database connection pooling, TLS pass-through |
| NGINX / OpenResty | Mature ecosystem, static asset speed, rich Lua scripting | Reload cost for config changes, single-threaded worker sensitivity | High-scale static caching, edge TLS termination, CDN nodes |
| Apache APISIX | Real-time dynamic updates via etcd, sub-ms sync, Radix tree | Complex ops (etcd quorum dependency), higher memory than raw NGINX | Microservice API Gateway, dynamic multi-tenant SaaS ingress |
| Kong Gateway | Rich enterprise plugin ecosystem, DB-less option | PostgreSQL scaling bottlenecks at high node counts if not DB-less | Enterprise developer portal, API governance, hybrid cloud ingress |
8. Practical Engineering Simulation: Edge Gateway & Token Bucket Router
The following self-contained TypeScript simulation models an edge router with an L4 load-balancing distributor, an L7 URL-prefix router with dynamic Radix matching, and a non-blocking token bucket rate-limiting filter.
/**
* Systems Notebook - Edge Gateway & Rate-Limiting Simulation
* Models L4 connection distribution, L7 routing, and non-blocking token bucket filters.
*/
interface Request {
id: string;
clientIp: string;
path: string;
method: 'GET' | 'POST';
timestampMs: number;
}
interface UpstreamTarget {
id: string;
host: string;
activeConnections: number;
weight: number;
isHealthy: boolean;
}
interface RouteRule {
prefix: string;
upstreamPool: string;
rateLimitQps: number;
burstCapacity: number;
}
interface RateLimitBucket {
tokens: number;
lastRefillMs: number;
}
class TokenBucketFilter {
private buckets: Map<string, RateLimitBucket> = new Map();
constructor(
private readonly qps: number,
private readonly capacity: number
) {}
public allowRequest(clientKey: string, nowMs: number): boolean {
let bucket = this.buckets.get(clientKey);
if (!bucket) {
bucket = { tokens: this.capacity, lastRefillMs: nowMs };
this.buckets.set(clientKey, bucket);
} else {
// Calculate token refill based on elapsed time
const elapsedSeconds = (nowMs - bucket.lastRefillMs) / 1000;
const tokensToAdd = elapsedSeconds * this.qps;
bucket.tokens = Math.min(this.capacity, bucket.tokens + tokensToAdd);
bucket.lastRefillMs = nowMs;
}
if (bucket.tokens >= 1.0) {
bucket.tokens -= 1.0;
return true; // Allowed
}
return false; // Rate-limited (HTTP 429)
}
}
class EdgeGateway {
private routes: RouteRule[] = [];
private upstreams: Map<string, UpstreamTarget[]> = new Map();
private rateLimiters: Map<string, TokenBucketFilter> = new Map();
public registerRoute(rule: RouteRule): void {
this.routes.push(rule);
// Sort descending by prefix length for longest-prefix match
this.routes.sort((a, b) => b.prefix.length - a.prefix.length);
this.rateLimiters.set(
rule.prefix,
new TokenBucketFilter(rule.rateLimitQps, rule.burstCapacity)
);
}
public registerUpstreamPool(poolName: string, targets: UpstreamTarget[]): void {
this.upstreams.set(poolName, targets);
}
/**
* L4 Least-Connection Load Balancing
*/
private selectTarget(poolName: string): UpstreamTarget | null {
const targets = this.upstreams.get(poolName);
if (!targets || targets.length === 0) return null;
const healthy = targets.filter((t) => t.isHealthy);
if (healthy.length === 0) return null;
// Pick target with minimum active connections
return healthy.reduce((min, curr) =>
curr.activeConnections < min.activeConnections ? curr : min
);
}
/**
* L7 Process Pipeline
*/
public handleRequest(req: Request): {
status: number;
upstreamNode?: string;
error?: string;
} {
// 1. Longest-Prefix L7 Route Matching
const matchedRoute = this.routes.find((r) => req.path.startsWith(r.prefix));
if (!matchedRoute) {
return { status: 404, error: 'Route Not Found' };
}
// 2. Token Bucket Rate Limiting (access_by_lua phase)
const limiter = this.rateLimiters.get(matchedRoute.prefix)!;
const allowed = limiter.allowRequest(req.clientIp, req.timestampMs);
if (!allowed) {
return { status: 429, error: 'Rate Limit Exceeded (HTTP 429)' };
}
// 3. Upstream Target Selection (L4 leastconn load balancer)
const target = this.selectTarget(matchedRoute.upstreamPool);
if (!target) {
return { status: 503, error: 'Service Unavailable (No Healthy Upstreams)' };
}
// Simulate connection acquisition
target.activeConnections++;
const responseNode = `${target.id} (${target.host})`;
// Simulate immediate release for this discrete tick
target.activeConnections--;
return { status: 200, upstreamNode: responseNode };
}
}
// -------------------------------------------------------------
// Verification Demonstration
// -------------------------------------------------------------
function runSimulation(): void {
const gateway = new EdgeGateway();
gateway.registerUpstreamPool('orders-cluster', [
{ id: 'node-1', host: '10.0.1.10:8080', activeConnections: 12, weight: 1, isHealthy: true },
{ id: 'node-2', host: '10.0.1.11:8080', activeConnections: 4, weight: 1, isHealthy: true },
{ id: 'node-3', host: '10.0.1.12:8080', activeConnections: 25, weight: 1, isHealthy: false } // Down
]);
gateway.registerRoute({
prefix: '/api/v1/orders',
upstreamPool: 'orders-cluster',
rateLimitQps: 5, // 5 requests per second
burstCapacity: 2 // Max burst of 2 tokens
});
const clientIp = '198.51.100.42';
let virtualTimeMs = 1000;
console.log('--- Phase 1: Burst of 3 concurrent requests ---');
for (let i = 1; i <= 3; i++) {
const res = gateway.handleRequest({
id: `req-${i}`,
clientIp,
path: '/api/v1/orders/create',
method: 'POST',
timestampMs: virtualTimeMs
});
console.log(`Req ${i}: Status ${res.status} | Routed To: ${res.upstreamNode ?? res.error}`);
}
console.log('\n--- Phase 2: Advance time by 500ms (Tokens refilled: 0.5 * 5 = 2.5) ---');
virtualTimeMs += 500;
const resAfterRefill = gateway.handleRequest({
id: 'req-4',
clientIp,
path: '/api/v1/orders/status',
method: 'GET',
timestampMs: virtualTimeMs
});
console.log(`Req 4: Status ${resAfterRefill.status} | Routed To: ${resAfterRefill.upstreamNode ?? resAfterRefill.error}`);
}
runSimulation();
Simulation Execution Output & State Transitions
--- Phase 1: Burst of 3 concurrent requests ---
Req 1: Status 200 | Routed To: node-2 (10.0.1.11:8080)
Req 2: Status 200 | Routed To: node-2 (10.0.1.11:8080)
Req 3: Status 429 | Routed To: Rate Limit Exceeded (HTTP 429)
--- Phase 2: Advance time by 500ms (Tokens refilled: 0.5 * 5 = 2.5) ---
Req 4: Status 200 | Routed To: node-2 (10.0.1.11:8080)
9. Observability & SRE Metrics
To maintain edge proxy health, SREs monitor four foundational golden signals:
+-------------------------------------------------------------------------------+
| KEY EDGE GATEWAY TELEMETRY |
+-------------------------------------------------------------------------------+
| Metric Name | Source / Exporter | SRE Target / Alert Rule |
+-------------------------+-----------------------+-----------------------------+
| Active Connections | HAProxy show info | Alert if active > 80% of |
| (current_conn) | NGINX stub_status | worker_connections limit |
| Upstream Response Time | $upstream_response_ | p99 < 50ms; p99.9 < 200ms |
| (p95, p99, p99.9) | time in access log | Spike indicates backend lag |
| Rate Limit Discards | APISIX / Kong metric | Spike indicates brute-force |
| (http_status 429) | prometheus plugin | or misconfigured client |
| etcd Watch Lag | APISIX etcd_sync_lag | Alert if lag > 500ms |
| | prometheus counter | indicates config drift |
+-------------------------+-----------------------+-----------------------------+
10. System Design Interview & Production Rubric
When presenting edge routing in a system design interview:
+------------------------------------------------------------------------------+
| LEVEL 3 (Senior Engineer) | LEVEL 4 (Staff / Principal Architect) |
+---------------------------------+--------------------------------------------+
| - Mentions NGINX or HAProxy as | - Distinguishes L4 zero-copy splice from |
| a reverse proxy box | L7 TLS termination and request parsing |
| - Configures round-robin and | - Designs dynamic etcd-backed control |
| basic SSL termination | planes to eliminate reload packet drop |
| - Relies on simple static IP | - Implements multi-tier rate limiting with |
| allow-lists | sharded shared memory and Redis fallback |
| - Unaware of reload packet drop | - Explains kernel SO_REUSEPORT and CPU core|
| under high SYN traffic | pinning to eliminate cache contention |
+---------------------------------+--------------------------------------------+
11. Exercises & Deliverables
- HAProxy vs NGINX Benchmarking:
- Deploy HAProxy in Layer 4 mode and NGINX in Layer 7 mode using Docker or Linux network namespaces.
- Run a wrk or vegeta benchmark with concurrent connections. Measure CPU utilization, context switch rate (
vmstat 1), and p99 latency.
- Hitless Reload Verification:
- Write a client script generating continuous HTTP requests at .
- Issue 50 successive reload signals to NGINX (
kill -HUP) and HAProxy (-xsocket handoff). Verify whether any TCP connections reset or return 502.
- Custom OpenResty Plugin:
- Implement an OpenResty Lua filter that extracts an HMAC-SHA256 signature from an incoming header, verifies it against a key stored in
lua_shared_dict, and injects a sanitized tenant ID header before proxying upstream.
- Implement an OpenResty Lua filter that extracts an HMAC-SHA256 signature from an incoming header, verifies it against a key stored in
View Reference Architecture & Worked Solutions (CORE-M36A-002)
1. Worked Solution — Exercise 1: HAProxy (L4) vs. NGINX (L7) Benchmarking ( Concurrent Connections)
- Data Path & Syscall Comparison:
- HAProxy (
mode tcp+option splice-auto): Executessplice(client_fd, pipe_w, len, SPLICE_F_MOVE | SPLICE_F_NONBLOCK)followed bysplice(pipe_r, backend_fd, len, SPLICE_F_MOVE | SPLICE_F_NONBLOCK). Payload pages remain pinned inside kernelsk_buffring memory; only DMA descriptor pointers move between socket buffers. Zero bytes are copied into user-space virtual memory, keeping L1/L2 CPU caches warm forebtreestick-table lookups. - NGINX (
httpL7 proxy): Executesepoll_wait()read(client_fd, user_buf)(kernel-to-user copy) HTTP state-machine header parsing and buffer chain allocation (ngx_buf_t)writev(upstream_fd, iov)(user-to-kernel copy).
- HAProxy (
- Expected Quantitative Telemetry under Connections (, payload):
- Memory Footprint: With tuned buffers ( kernel socket + user buffer), connections consume RAM—confirming event-driven per-connection overhead.
- CPU & Context Switches (
vmstat 1): With worker threads/processes pinned 1:1 to physical cores (cpu-mapin HAProxy,worker_cpu_affinityin NGINX), voluntary context switches (cs) remain low (, driven purely byepoll_waitwakeups). HAProxy L4 consumes less CPU bandwidth than L7 proxying due to zero-copy splicing and bypassing HTTP header tokenization, yielding proxy latency of (L4) vs. (L7).
2. Worked Solution — Exercise 2: Hitless Reload Verification (NGINX SIGHUP vs. HAProxy -x Socket Handoff)
- Why Traditional Reloads Drop Packets Under High SYN Velocity:
- When
SO_REUSEPORTis enabled, the Linux kernel hashes incomingSYNpackets across the set of active listening socket file descriptors. During a standardkill -HUPreload, old workers close their listening FDs while new workers bind freshSO_REUSEPORTsockets. Any connection that completed its 3-way handshake (or is mid-handshake) inside an old listening socket’sacceptqueue at the moment that FD closes is sent a TCPRST(or dropped) unless kernelBPF_SOCK_OPSreuseport groups drain it cleanly. At with rapid reloads, a drain gap per reload risks resetting handshakes.
- When
- HAProxy Hitless Reload Configuration & Mechanics:
- Configure the master-worker stats socket in
haproxy.cfg:global master-worker stats socket /var/run/haproxy-master.sock mode 600 level admin expose-fd listeners - Reload via
haproxy -W -db -f /etc/haproxy/haproxy.cfg -x /var/run/haproxy-master.sock -sf $(cat /var/run/haproxy.pid). - The new HAProxy process connects to
/var/run/haproxy-master.sockand receives the exact open listening file descriptors via UNIX domain socketsendmsg(2)withSCM_RIGHTSancillary data. Because the kernel socket structs and theirSYN/accept queues are never closed or re-allocated, zero packets are dropped across all 50 reloads.
- Configure the master-worker stats socket in
3. Worked Solution — Exercise 3: Custom Non-Blocking OpenResty HMAC-SHA256 Plugin
-- nginx.conf (http block):
-- lua_shared_dict tenant_keys 10m;
-- init_worker_by_lua_block {
-- ngx.shared.tenant_keys:set("tenant_acme", "super-secret-signing-key-256")
-- }
-- access_by_lua_block implementation:
local resty_hmac = require("resty.hmac")
local resty_str = require("resty.string")
local headers = ngx.req.get_headers()
local tenant_id = headers["X-Client-Tenant"]
local signature = headers["X-Signature-HMAC"]
if not tenant_id or not signature then
ngx.status = ngx.HTTP_UNAUTHORIZED
ngx.say('{"error":"Missing X-Client-Tenant or X-Signature-HMAC header"}')
return ngx.exit(ngx.HTTP_UNAUTHORIZED)
end
-- 1. O(1) lock-free/spinlock lookup from POSIX shared memory slab (zero network I/O)
local secret = ngx.shared.tenant_keys:get(tenant_id)
if not secret then
ngx.status = ngx.HTTP_FORBIDDEN
ngx.say('{"error":"Unknown tenant identity"}')
return ngx.exit(ngx.HTTP_FORBIDDEN)
end
-- 2. Read request body non-blockingly and compute HMAC-SHA256
ngx.req.read_body()
local body_data = ngx.req.get_body_data() or ""
local signing_input = ngx.var.request_method .. ":" .. ngx.var.uri .. ":" .. body_data
local hmac_sha256 = resty_hmac:new(secret, resty_hmac.ALGOS.SHA256)
hmac_sha256:update(signing_input)
local expected_hex = resty_str.to_hex(hmac_sha256:final())
-- 3. Constant-time byte comparison to prevent timing side-channel attacks
if #signature ~= #expected_hex then
return ngx.exit(ngx.HTTP_UNAUTHORIZED)
end
local diff = 0
for i = 1, #expected_hex do
diff = bit.bor(diff, bit.bxor( string.byte(signature, i), string.byte(expected_hex, i) ))
end
if diff ~= 0 then
return ngx.exit(ngx.HTTP_UNAUTHORIZED)
end
-- 4. Sanitize and inject verified tenant header for upstream microservice
ngx.req.clear_header("X-Signature-HMAC")
ngx.req.set_header("X-Verified-Tenant-ID", tenant_id)
- Event-Loop Safety Invariant: Every operation (
ngx.shared.DICT:get,ngx.req.read_body, OpenSSL C-FFIresty.hmac) executes either in-memory in microseconds or yields via OpenResty’sepoll-integrated coroutine scheduler, guaranteeing zero worker event-loop stalls.
12. Comprehensive Architectural Summary
Modern edge routing relies on specialized architectural divisions:
- HAProxy reigns at Layer 4 transport proxying due to its multi-threaded event engine, zero-copy kernel
splice()data relay, and lightning-fastebtreestick tables. - NGINX and OpenResty dominate Layer 7 web serving and static content due to battle-tested master-worker isolation and LuaJIT non-blocking coroutines.
- Apache APISIX and Kong represent the modern programmable API gateway tier, with APISIX leveraging etcd v3 MVCC HTTP/2 streaming to achieve zero-reload real-time configuration synchronization across globally distributed edge fleets.
13. Authoritative References & Version Context
- Igor Sysoev (2004–2024), “NGINX Architecture Overview & Internal Event Processing” — nginx.org/en/docs/
- Willy Tarreau (2000–2024), “HAProxy Architecture and Performance Tuning Guide” — haproxy.org
- Kong Gateway Documentation (v3.5+) — Architecture and Declarative DB-less Configuration — docs.konghq.com/gateway/latest/
- Version Context: Edge gateways reflect NGINX 1.26 LTS, HAProxy 2.8 / 3.0 LTS, and Kong Gateway 3.5+ LTS enterprise deployments.