1. Problem and Motivation
High-concurrency reservation scenarios—such as stadium concert tickets (Taylor Swift Eras Tour, Glastonbury), limited-edition sneaker drops, airline seat allocations, and Black Friday flash sales—represent the ultimate concurrency challenge in distributed system design:
+---------------------------------------------------------------------------------------------------+
| THE FLASH SALE PROBLEM SPACE |
+---------------------------------------------------------------------------------------------------+
| 1. Extreme Traffic Asymmetry: Idle traffic of 50 RPS surges instantaneously to 200,000 RPS at |
| 10:00:00 AM (a 4,000x surge within 200 milliseconds). |
| 2. Extreme Resource Scarcity: 50,000 tickets available for 3,000,000 competing buyers. |
| 3. Zero-Overselling Constraint: Selling 50,001 tickets for a 50,000-seat stadium causes legal, |
| financial, and physical security violations. Overbooking is strictly forbidden. |
| 4. Low Latency Under Pressure: Frustrated users furiously refresh browsers, multiplying request |
| rates exponentially if response latency degrades beyond 1 second. |
| 5. Bot & Scalper Warfare: Sophisticated automated headless browser farms and sniper scripts |
| probe API endpoints to corner inventory within 50 milliseconds of release. |
| 6. The Payment Delay Window: Payment processing via Stripe/Adyen takes 2–5 seconds, during which |
| reserved inventory must be held exclusively without blocking the rest of the cluster. |
+---------------------------------------------------------------------------------------------------+
2. Mental Model & Architectural Overview
To survive flash surges without over-provisioning thousands of idle database instances, the architecture decouples traffic handling into four distinct tiers:
1. 100,000+ Concurrent Users & Virtual Waiting Room (Edge CDN)
Establishes the ingress and structural boundary across 100,000+ Concurrent Users & Virtual Waiting Room (Edge CDN): Browsers & Sniper Bots hitting site GET /events/taylor-swift-eras/tickets; Intercepts 99.5% of traffic before origin; Assigns cryptographically signed JWT.
2. Reservation Service & Redis Lua Script: Atomic Hold
Coordinates core data-plane execution and state transitions through Reservation Service & Redis Lua Script: Atomic Hold: Validates signed queue ticket & user session POST /api/v1/reserve { event_id, qty: 2 }; Executes atomically in single Redis thread: stock = redis.call('GET', 'stock:event_1') if tonumber(stock) >= qty then redis.call('DECRBY', 'stock:event_1', qty) redis.call('SETEX', 'hold:res_123', 600, qty) return 1 -- SUCCESS else return 0 -- SOLD OUT; Stock is decremented; temporary lease created. User receives reservation ID: "res_123" 10:00 countdown timer starts in UI.
3. External Payment Gateway & Asynchronous Webhook Settlement
Enforces durability, state convergence, and output verification across External Payment Gateway & Asynchronous Webhook Settlement: Stripe / Adyen processing (1 - 3 sec) Zero hold on database row locks!; Payment Intent: CHARGED; POST /webhooks/stripe/payment_intent Verified via HMAC-SHA256 signature Idempotent processing via event_id.
The Architectural Tiers:
- Edge Traffic Gate & Virtual Waiting Room (VWR):
- Intercepts 100% of raw user traffic at the CDN Edge (Cloudflare / CloudFront).
- Assigns each user a cryptographically signed HMAC token containing queue position and timestamp.
- Admits a smooth, calibrated outflow (e.g., exactly 500 users per second) into the origin application gateway, matching the origin cluster’s validated throughput ceiling.
- Atomic In-Memory Reservation Tier (Redis Cluster):
- Evaluates inventory requests entirely in RAM via single-threaded Redis Lua scripts.
- Executes atomic check-and-decrement: if stock , decrements stock and creates a 10-minute hold lease (
hold:res_123withEX 600). - If stock is exhausted, immediately returns
HTTP 409 Sold Outin —completely shielding backend databases from failed probes.
- Asynchronous Payment Saga & Checkout:
- The user has 10 minutes to complete payment.
- When payment succeeds, payment gateway webhooks trigger an asynchronous event confirming the order in PostgreSQL and finalizing the inventory transfer.
- Automated Inventory Reclamation:
- If 10 minutes elapse without payment confirmation, the Redis lease expires, triggering an atomic inventory increment back to the available pool.
3. Terminology & Core Mechanics
- Overselling (Double-Booking): A critical race condition where two concurrent transactions read available stock as positive and both decrement it, causing inventory to drop below zero.
- Row-Lock Contention: When multiple database connections queue for an exclusive write lock on the same row, degrading transaction throughput from thousands per second to single digits.
- Virtual Waiting Room: A FIFO queueing layer situated between clients and origin servers that buffers traffic spikes at the edge and throttles admission.
- Inventory Hold Lease: A temporary exclusive lock on an inventory item granted to a user for a fixed time window (typically 5 to 10 minutes) while they complete checkout.
- Hot-Row Sharding: Partitioning a single hot inventory balance across sub-counters to allow concurrent updates without lock serialization.
- Two-Phase Reservation Saga: Decoupling the reservation action (
HELD) from the financial settlement action (CONFIRMED) to ensure external payment delays never block inventory locks.
4. Internal Workings: Atomic Redis Lua & Waiting Room Invariants
1. The Catastrophic Database Row-Lock Collapse
Why can we not simply use a relational database with transactions?
BEGIN TRANSACTION;
SELECT stock FROM tickets WHERE event_id = 42 FOR UPDATE; -- Acquires Exclusive Row Lock!
-- Application calculates if stock >= qty
UPDATE tickets SET stock = stock - 2 WHERE event_id = 42;
COMMIT;
Let:
- concurrent requests arriving in 1 second.
- (network round-trip + query parse + index lock + WAL write).
Because all transactions require the same row lock, they serialize completely:
Time required to process 20,000 requests sequentially:
While requests wait in line, database connection pools exhaust, thread context switching consumes 100% of CPU, lock wait timeouts cascade, and the database crashes.
2. The Atomic Redis Lua Solution
By delegating the critical section to Redis, we eliminate distributed locking, database row locks, and network round-trip interleaving:
-- KEYS[1]: stock key (e.g., "{event_42}:stock")
-- KEYS[2]: hold hash key (e.g., "{event_42}:hold:res_123")
-- KEYS[3]: expiry ZSET key (e.g., "{event_42}:holds_expiry")
-- ARGV[1]: requested quantity (e.g., 2)
-- ARGV[2]: hold lease duration in seconds (e.g., 600)
-- ARGV[3]: user ID
-- ARGV[4]: current Unix epoch seconds (passed deterministically by caller)
-- ARGV[5]: reservation ID ("res_123")
if redis.call('EXISTS', KEYS[2]) == 1 then
local existing_qty = tonumber(redis.call('HGET', KEYS[2], 'qty'))
return {1, tonumber(redis.call('GET', KEYS[1]) or '0'), existing_qty} -- Idempotent replay
end
local stock = redis.call('GET', KEYS[1])
if not stock then
return {err = "EVENT_NOT_FOUND"}
end
local current_stock = tonumber(stock)
local requested_qty = tonumber(ARGV[1])
local now_sec = tonumber(ARGV[4])
local expire_at = now_sec + tonumber(ARGV[2])
if current_stock >= requested_qty then
-- 1. Decrement available stock atomically
redis.call('DECRBY', KEYS[1], requested_qty)
-- 2. Record hold metadata and index expiration deadline in ZSET
-- (Note: hash tag {event_42} co-locates all three keys on the same Redis Cluster slot)
redis.call('HSET', KEYS[2],
'user_id', ARGV[3],
'qty', requested_qty,
'status', 'HELD',
'created_at', now_sec,
'expires_at', expire_at
)
redis.call('ZADD', KEYS[3], expire_at, ARGV[5])
return {1, current_stock - requested_qty} -- SUCCESS: [status, remaining]
else
return {0, current_stock} -- SOLD OUT
end
Why Index Deadlines in a ZSET Instead of Relying Solely on Redis EXPIRE Keyspace Events?
Redis Pub/Sub Keyspace Notifications (__keyevent@0__:expired) are fire-and-forget (events are dropped if a consumer disconnects) and only emit the key name after the hash fields (qty) have already been deleted! Recording expire_at in {event_42}:holds_expiry allows a deterministic Lua reaper script (ZRANGEBYSCORE ... -inf now) to read qty from {event_42}:hold:<id>, transition status to EXPIRED once, and execute INCRBY without ever losing a single seat.
Because Redis executes Lua scripts as a single atomic operation, no other command can execute between the GET and the DECRBY. This guarantees zero overselling at memory speeds of per operation.
5. Mathematical / Performance Model
Let us mathematically analyze admission rates, waiting room sizing, and hot-row lock scaling.
1. Calibrated Admission Rate Derivation
Let:
- : Maximum sustained write capacity of origin database cluster ()
- : Safety buffer factor ( to to prevent database saturation)
- : Incoming spike rate ()
- : Admission rate through the Virtual Waiting Room
To guarantee zero database queuing:
Fraction of traffic absorbed and buffered by the Edge Waiting Room:
The edge waiting room absorbs of traffic, keeping origin systems operating at a calm, predictable baseline.
By Little’s Law (), if enter the checkout flow and spend an average dwell time of () selecting seats and completing payment, the origin tier maintains a bounded steady-state concurrency of:
If of admitted sessions successfully reserve an average of tickets from a -seat stadium (), primary inventory depletes in (), after which the waiting room transitions remaining queued fans to standby mode for reclaimed expired holds.
2. Hot-Row Inventory Sharding
When selling 100,000 general admission tickets, even a single Redis key can become a CPU core bottleneck under 150,000 RPS.
We apply Inventory Sub-Partitioning (Sharding across sub-keys):
[ Total Stock: 100,000 tickets ]
Divided across K = 10 independent Redis keys:
stock:event_42:slot_0 = 10,000
stock:event_42:slot_1 = 10,000
...
stock:event_42:slot_9 = 10,000
When a user requests tickets, the reservation service hashes the user’s ID to select a slot:
If slot_i is depleted, the client attempts the adjacent slot (slot_i + 1) \pmod K.
This increases theoretical Redis write throughput by , distributing updates across multiple CPU cores and cluster nodes.
6. Real Implementation & Systems Comparison
| Architecture Feature | Naive Relational DB | Redlock Distributed Lock | Redis Lua + Edge VWR (Industry Standard) |
|---|---|---|---|
| Max Safe Ingress QPS | (Edge Buffered) | ||
| P99 Reservation Latency | (Lock timeouts) | (Multi-node locks) | (Single-threaded Lua RAM) |
| Overselling Risk | High under isolation bugs | Low (Clock drift edge cases) | Zero (Mathematically guaranteed) |
| Origin Server Load | Extreme (All users hit DB) | High (All users hit backend) | Minimal (Only admitted users hit backend) |
| User Experience | 504 Gateway Timeouts, blank screens | Error popups on lock failure | Transparent queue position countdown |
7. Production Code Engine: Flash Sale Engine
The following complete TypeScript implementation demonstrates:
- The Atomic Inventory Hold & Release Engine (simulating Redis Lua execution).
- The Virtual Waiting Room Token Issuer & Validator.
- The Checkout Payment Saga with automatic timeout lease expiration.
/**
* Production-Grade Flash Sale & Ticket Reservation Engine
* Demonstrates atomic hold leases, queue admission, and checkout saga.
*/
export interface ReservationLease {
reservationId: string;
userId: string;
eventId: string;
quantity: number;
expiresAt: number; // Unix timestamp
status: 'HELD' | 'CONFIRMED' | 'EXPIRED' | 'CANCELLED';
}
export class FlashSaleEngine {
private stock: Map<string, number> = new Map();
private leases: Map<string, ReservationLease> = new Map();
constructor() {}
public initializeEvent(eventId: string, initialStock: number): void {
this.stock.set(eventId, initialStock);
}
public getAvailableStock(eventId: string): number {
return this.stock.get(eventId) ?? 0;
}
/**
* Atomic Reservation via Redis Lua Simulation
* Checks stock, decrements, and creates hold lease in one atomic step.
*/
public atomicHold(
eventId: string,
userId: string,
quantity: number,
leaseTtlSeconds = 600
): { success: boolean; lease?: ReservationLease; error?: string } {
const currentStock = this.stock.get(eventId);
if (currentStock === undefined) {
return { success: false, error: 'EVENT_NOT_FOUND' };
}
// Atomic Check and Decrement
if (currentStock >= quantity) {
this.stock.set(eventId, currentStock - quantity);
const reservationId = `res_${eventId}_${userId}_${Date.now()}`;
const lease: ReservationLease = {
reservationId,
userId,
eventId,
quantity,
expiresAt: Date.now() + leaseTtlSeconds * 1000,
status: 'HELD',
};
this.leases.set(reservationId, lease);
return { success: true, lease };
}
return { success: false, error: 'SOLD_OUT' };
}
/**
* Confirms payment and permanently finalizes reservation.
*/
public confirmReservation(reservationId: string): { success: boolean; error?: string } {
const lease = this.leases.get(reservationId);
if (!lease) {
return { success: false, error: 'RESERVATION_NOT_FOUND' };
}
if (lease.status !== 'HELD') {
return { success: false, error: `CANNOT_CONFIRM_STATUS_${lease.status}` };
}
if (Date.now() > lease.expiresAt) {
lease.status = 'EXPIRED';
this.reclaimInventory(lease);
return { success: false, error: 'RESERVATION_LEASE_EXPIRED' };
}
lease.status = 'CONFIRMED';
return { success: true };
}
/**
* Background Worker / Keyspace notification: Reclaims expired inventory.
*/
public sweepExpiredLeases(): number {
const now = Date.now();
let reclaimedCount = 0;
for (const [_, lease] of this.leases.entries()) {
if (lease.status === 'HELD' && now > lease.expiresAt) {
lease.status = 'EXPIRED';
this.reclaimInventory(lease);
reclaimedCount++;
}
}
return reclaimedCount;
}
private reclaimInventory(lease: ReservationLease): void {
const current = this.stock.get(lease.eventId) ?? 0;
this.stock.set(lease.eventId, current + lease.quantity);
}
}
/**
* Edge Virtual Waiting Room Token Manager
*/
export class VirtualWaitingRoom {
private currentAdmissionSeq = 0;
private totalEnqueued = 0;
constructor(
private readonly admissionRatePerSecond: number,
private readonly secretKey: string
) {}
public enqueueUser(userId: string): { queuePosition: number; estimatedWaitSeconds: number } {
this.totalEnqueued++;
const position = this.totalEnqueued - this.currentAdmissionSeq;
const waitTime = Math.ceil(position / this.admissionRatePerSecond);
return { queuePosition: position, estimatedWaitSeconds: waitTime };
}
public advanceAdmission(secondsElapsed: number): void {
const admittedCount = secondsElapsed * this.admissionRatePerSecond;
this.currentAdmissionSeq = Math.min(this.totalEnqueued, this.currentAdmissionSeq + admittedCount);
}
public isAdmitted(queueNumber: number): boolean {
return queueNumber <= this.currentAdmissionSeq;
}
}
function runDemo(): void {
console.log("=== Flash Sale & Virtual Waiting Room Architecture ===");
const flashSale = new FlashSaleEngine();
const waitingRoom = new VirtualWaitingRoom(50, "edge-hmac-secret-v1");
flashSale.initializeEvent("event-taylor-swift", 100);
console.log("Initialized Flash Sale Event: 100 VIP tickets available");
const q1 = waitingRoom.enqueueUser("user-alice");
const q2 = waitingRoom.enqueueUser("user-bob");
console.log(`Alice queue position: ${q1.queuePosition}, Bob queue position: ${q2.queuePosition}`);
waitingRoom.advanceAdmission(30);
const aliceHold = flashSale.atomicHold("event-taylor-swift", "user-alice", 2, 60);
console.log(
`Alice atomic hold: ${aliceHold.success ? "HELD" : "FAILED"}, leaseId=${aliceHold.lease?.reservationId}`
);
console.log(`Remaining stock after Alice hold: ${flashSale.getAvailableStock("event-taylor-swift")} tickets`);
if (aliceHold.lease) {
const confirmation = flashSale.confirmReservation(aliceHold.lease.reservationId);
console.log(`Alice reservation confirmed: ${confirmation.success}`);
}
// Simulate an abandoned hold whose TTL already expired (-1s)
const bobHold = flashSale.atomicHold("event-taylor-swift", "user-bob", 4, -1);
console.log(`Bob hold created (stock=${flashSale.getAvailableStock("event-taylor-swift")}), leaseId=${bobHold.lease?.reservationId}`);
const reclaimed = flashSale.sweepExpiredLeases();
console.log(`Swept ${reclaimed} expired lease(s); restored stock: ${flashSale.getAvailableStock("event-taylor-swift")} tickets`);
}
runDemo();
8. Hotspot Mitigation & Scalper Defense
1. Bot & Sniper Scalper Mitigation
- Proof-of-Work (PoW) Challenge: Before placing a user into the waiting room, the client browser must solve a cryptographic SHA-256 hash puzzle requiring 1.5 seconds of local CPU computation. This makes running 100,000 headless Puppeteer bot instances economically non-viable.
- Signed Admission Cookies: Admitted users receive a cryptographically signed HMAC-SHA256 cookie (
queue_ticket=user_123:exp_1761899120:signature). Origin gateways reject any checkout request lacking a valid, unexpired ticket.
2. Preventing Inventory Held Hostage Attacks
Malicious actors write scripts that reserve all available inventory tickets and purposefully abandon checkout, denying legitimate fans the opportunity to purchase.
- Strict Cart Limits: Maximum 4 tickets per authenticated user account and payment instrument.
- Gradual Inventory Releases: Rather than releasing all 50,000 tickets at 10:00:00 AM, the system releases inventory in waves of 10,000 every 15 minutes, diluting the impact of any coordinated scalper campaign.
9. Failure Modes & Operational Perspective
| Failure Scenario | Manifestation | Production Mitigation |
|---|---|---|
| Redis Node Crash During Sale | In-memory stock counter lost. | Redis cluster with Multi-AZ replication and appendonly yes (AOF fsync every 1 sec); fallback to replica node via Redis Sentinel in . |
| Payment Gateway Timeout | User card charged, but webhook dropped by network partition. | Payment webhook processor uses Idempotency Keys with exponential retries; background reconciliation job polls Stripe /events API for unlinked charges. |
| User Abandons Checkout Page | Tickets remain in HELD state forever. | Strict 10-minute Redis TTL leases; automated sweeper increments available stock upon lease expiration. |
| Waiting Room Token Forgery | Attackers forge admission tokens to bypass queue. | Tokens signed using asymmetric Ed25519 or HMAC-SHA256 with key rotation; origin gateway verifies signature in . |
10. Knowledge Check
Review the core flash sale architectural principles, then complete the 4-question interactive assessment quiz at the end of this lesson:
- Pessimistic Locking Bottleneck: Explain why relational database row locks serialize updates and cause connection pool starvation under flash traffic.
- Atomic Lua Execution: Trace how single-threaded Redis Lua scripts execute check-and-decrement operations in sub-millisecond RAM with zero race conditions.
- Virtual Waiting Room Sizing: Calculate how edge admission throttling shields origin databases from being overwhelmed.
- Lease Expiration Reclamation: Describe the lifecycle of a reservation from
HELDtoCONFIRMEDorEXPIRED.
11. Practical Exercises & Deliverables
Exercise 1: Implement Redis Lua Sharded Inventory
Problem: A single Redis key maxes out at 80,000 operations/sec on a single CPU core.
Deliverable:
- Implement an inventory manager that shards a total stock of 50,000 items across 8 distinct Redis keys (
stock:event_1:shard_0throughshard_7). - Write a client routing algorithm that attempts reservation on a hashed shard and falls back to adjacent shards if the primary shard is exhausted.
- Benchmark the throughput improvement against a single-key baseline under 10,000 concurrent requests.
Exercise 2: Build a Virtual Waiting Room Simulator
Problem: Test how your origin servers perform when admitting users at varying rates.
Deliverable:
- Use the
VirtualWaitingRoomTypeScript class from Section 7 to simulate a queue with 50,000 users. - Advance time at 1-second intervals admitting 200 users/second.
- Track and graph: peak queue length, average user wait time, and origin server request rate over time.
View Reference Architecture & Worked Solutions (APP-A07-001)
1. Reference Solution — Exercise 1: Implement Redis Lua Sharded Inventory
- 8-Shard Partitioning & Ring-Fallback Routing:
- Partition total tickets across independent Redis Cluster hash-tagged keys (
{event_1:shard_0}:stockthrough{event_1:shard_7}:stock), initializing each shard with .
export class ShardedFlashSaleInventory { private readonly shards = new Array<number>(8); constructor(totalStock = 50_000) { const base = Math.floor(totalStock / 8); for (let i = 0; i < 8; i++) { this.shards[i] = i === 0 ? base + (totalStock % 8) : base; } } public tryReserve(userId: string, qty: number): { success: boolean; shardId?: number } { // Deterministic starting slot from userId hash let hash = 0; for (let i = 0; i < userId.length; i++) hash = (hash * 31 + userId.charCodeAt(i)) >>> 0; const startSlot = hash % 8; // Probe primary slot, then fall back clockwise across remaining slots for (let offset = 0; offset < 8; offset++) { const slot = (startSlot + offset) % 8; if (this.shards[slot] >= qty) { this.shards[slot] -= qty; // Atomic Lua check-and-decrement on shard `slot` return { success: true, shardId: slot }; } } return { success: false }; } } - Partition total tickets across independent Redis Cluster hash-tagged keys (
- Throughput Scaling: Because each
{event_1:shard_i}key hashes to a separate Redis Cluster primary node/core (), sharding across slots raises the aggregate atomic reservation ceiling from to while preserving strict zero-overselling invariants on every shard.
2. Reference Solution — Exercise 2: Build a Virtual Waiting Room Simulator
- Quantitative Waiting Room Trajectory ( Users, ):
- Peak Queue Length: At , all fans arrive at the edge waiting room:
- Total Queue Drainage Time ():
- Wait Time Distribution:
- First batch (positions ): .
- Median user (position ): .
- Last user (position ): .
- Average User Wait Time: .
- Origin Server Request Rate: Throughout the entire event, the origin checkout tier receives a flat, rock-steady admission rate of ( spike reduction vs. the initial surge).
12. Summary & Architectural Decision Guide
- Absorb Surges at the Edge Before Origin: Gate drops behind an Edge Virtual Waiting Room issuing cryptographically signed admission tokens calibrated to of validated origin checkout capacity.
- Replace Hot SQL Row Locks with Atomic Redis Lua: Execute check-and-decrement plus hold creation inside a single-threaded Redis Lua script co-located via hash tags (
{event_id}), and shard single-SKU general admission counters across slots when exceeding single-core Redis limits. - Never Hold Locks Across External PSP Calls: Decouple inventory reservation (
HELDlease for 600s) from asynchronous Stripe/Adyen payment capture via a two-phase checkout saga. - Index Lease Deadlines Deterministically: Track hold expirations in a Redis Sorted Set swept by an atomic Lua reaper rather than relying on lossy keyspace-expiration Pub/Sub events.
13. Authoritative References & Version Context
- Redis Official Documentation — Programmability & Atomic Lua Scripting: Redis Lua API & Script Atomicity (single-threaded
EVAL/EVALSHAexecution guarantees and Redis Cluster{hash_tag}key co-location; baseline Redis 7.2+). - Cloudflare Waiting Room Documentation — Architecture & Signed Queue Tokens: How Cloudflare Waiting Room Queues Traffic at the Edge (edge admission throttling and cryptographic queue cookie verification).
- Hector Garcia-Molina & Kenneth Salem (ACM SIGMOD 1987): Sagas (foundational compensating-transaction formulation for long-lived checkout and reservation workflows).
- PostgreSQL Official Documentation — Explicit Row-Level Locking (
FOR UPDATE): PostgreSQL 16/17 Explicit Locking (row-level lock queuing semantics and contention behavior; baseline stack: PostgreSQL 16+, Apache Kafka 3.7+).