Practice Library & Incident Runbooks

Periodic stop-the-world garbage collection pauses combined with lock contention on shared in-memory connection pools.

Migrate to non-blocking ring buffers (Disruptor pattern), tune GC allocation rates, and implement client request hedging.

Application code holding database transactions open while making synchronous external HTTP calls to third-party payment APIs.

Enforce transactional outbox pattern; move external RPCs strictly outside the database transaction boundary.

Clients executing uncoordinated aggressive exponential retries with zero jitter, creating a self-inflicted DDoS retry storm.

Implement circuit breakers (hystrix/envoy), full jitter backoff schedules, and adaptive load shedding at ingress.

Data skew on message key: a celebrity account generated 80% of total write events mapped to a single partition.

Salt high-cardinality keys with random sub-partition suffixes (e.g. key_0..key_9) or use round-robin dispatch.

Frequent cell overwrites and deletions created millions of tombstones; read repair triggers heavy full-table scanning.

Tune `tombstone_failure_threshold`, decrease gc_grace_seconds for short-lived records, and run scheduled incremental repairs.

Asymmetric network partition isolated the leader in a minority 2-node cluster segment while 3 nodes lacked heartbeat quorum.

Configure odd-numbered cluster nodes (5 or 7), verify physical rack/zone affinity, and adjust leader election jitter.

Low virtual node factor (V=1) caused uneven token distribution on 32-bit hash ring; hot product key pinned to Node 4.

Increase virtual node factor to V ≥ 150; implement client-side local caching for top 0.1% hot keys.

Large Eden space allocations of short-lived byte arrays promoting directly to Old Generation, triggering 8-second G1 pauses.

Allocate buffers off-heap (DirectByteBuffers) or enable Shenandoah / ZGC low-latency concurrent garbage collectors.

Concurrent transactions acquiring locks on Shard A and Shard B in reverse order without global lock hierarchy.

Order shard lock acquisitions strictly by global entity ID ascending; replace cross-shard 2PC with asynchronous Sagas.

Fixed TTL cache expiration without probabilistic early refreshing or single-flight locking.

Implement XFetch probabilistic early expiration (`-beta * log(rand())`) and in-memory single-flight coalescing.

Upstream microservices failed to send HTTP keepalive responses, leaving 65,000 idle sockets in TIME_WAIT state.

Configure upstream keepalive connection pools, tune `tcp_tw_reuse`, and raise system `nofile` limits in limits.conf.

Static pre-allocated contiguous key-value (KV) cache buffers fragmented VRAM, leaving 70% of memory unusable.

Deploy PagedAttention (vLLM) to page KV caches into non-contiguous physical blocks with dynamic copy-on-write.

Asymmetric batch size across workers caused Rank 12 to miss a Ring-AllReduce barrier while other ranks waited indefinitely.

Set strict NCCL_ASYNC_ERROR_HANDLING=1 and ensure uniform token masking across all distributed training ranks.

Vector index (HNSW) build batch job throttled under write pressure, leaving millions of vectors unindexed in raw memory.

Decouple ingestion WAL from query segment compaction; deploy hybrid BM25 lexical + dense ANN search.

Asynchronous cross-datacenter replication delay exceeded client HTTP redirect latency; session stickiness was missing.

Pin recently mutated user reads to primary region via session cookie token for 5 seconds following writes.