Distributed Transactions: 2PC, 3PC, Sagas & TCC

1. The Problem & Why It Matters

In a monolithic architecture backed by a single relational database, maintaining transaction atomicity is straightforward:

BEGIN TRANSACTION;
  UPDATE accounts SET balance = balance - 100 WHERE id = 'alice';
  UPDATE accounts SET balance = balance + 100 WHERE id = 'bob';
COMMIT;

The database storage engine locks the rows, appends commit records to the write-ahead log (WAL), and guarantees Atomicity, Consistency, Isolation, and Durability (ACID). If the database crashes mid-flight, recovery rollback routines clean up uncommitted changes.

In modern distributed systems, data is fragmented across microservices, separate database engines, and third-party SaaS APIs:

• An e-commerce checkout must:
  1. Deduct customer loyalty credits in PostgreSQL.
  2. Reserve warehouse inventory in a MySQL shard.
  3. Authorize payment through Stripe’s external HTTP API.
  4. Generate a shipping label via FedEx.

If Stripe succeeds but FedEx rejects the shipping address, you cannot simply execute ROLLBACK. Stripe already charged the credit card, and PostgreSQL already committed the loyalty credit deduction!

This lesson explores how distributed systems solve atomic commitments across independent nodes: from the strict locking of Two-Phase Commit (2PC) to the eventual consistency of Sagas and the resource reservations of Try-Confirm-Cancel (TCC).

2. Intuitive Mental Model

Imagine booking a vacation requiring an Airline Ticket, a Hotel Room, and a Rental Car:

• Two-Phase Commit (2PC): You call all three vendors simultaneously. You ask: “Can you hold this seat/room/car?” (Prepare). All three say “Yes” and freeze the reservations. You then tell all three: “Book it!” (Commit). If your phone battery dies while they are on hold, all three vendors refuse to rent those seats/rooms to anyone else, waiting indefinitely for your call (The 2PC Blocking Hazard).
• The Saga Pattern: You book and pay for the flight immediately. Then you book and pay for the hotel. When you try to book the rental car, all cars are sold out! You cannot undo the past, so you execute Compensating Actions: you cancel the hotel (getting a refund) and cancel the flight (paying a cancellation fee).
• Try-Confirm-Cancel (TCC): You call the hotel and ask them to put a $100 pre-authorization hold on your card (Try). You do the same for the car. Once all pre-authorizations succeed, you call them back to finalize the charges (Confirm). If one vendor fails, you call the others to release the holds (Cancel).

SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED SYSTEMS

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. 2PC Phase 1: Prepare & Lock Acquisition

Coordinator broadcasts PREPARE? to DB Node A and B. Both nodes acquire exclusive row locks in local databases and reply YES.

Architectural Invariant: Isolation Cost: While in PREPARED state, exclusive locks block all concurrent readers and writers on those rows.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. Saga Forward Steps: T1 & T2 Commit Locally

Saga Orchestrator invokes Step 1 (Reserve Seat) and Step 2 (Charge Card). Each step commits immediately in its respective service database without holding cross-service locks.

Architectural Invariant: No Global Lock: Eliminates distributed locking. Downside: other transactions can observe intermediate committed states (dirty reads).

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Step T3 Failure & Backward Compensations (C2, C1)

T3 (Book Hotel) fails due to inventory depletion. Orchestrator triggers semantic backward compensations: C2 (Refund Card) and C1 (Release Seat) to restore system consistency.

Architectural Invariant: Eventual Consistency: The system guarantees that every completed forward step is eventually reversed if the overall workflow fails.

3. Architectural Mechanisms & Concrete Topologies

3.1 Two-Phase Commit (2PC) Mechanics

2PC coordinates atomic commitment across a Coordinator and $N$ Participants:

COORDINATOR                         PARTICIPANT 1                PARTICIPANT 2
    |                                     |                            |
    |---- 1. PREPARE -------------------->|                            |
    |---- 1. PREPARE ------------------------------------------------->|
    |                                     | [Locks Rows & Writes WAL]  | [Locks Rows & Writes WAL]
    |<--- 2. VOTE_COMMIT (YES) -----------|                            |
    |<--- 2. VOTE_COMMIT (YES) ----------------------------------------|
    |
[Writes COMMIT to WAL]
    |
    |---- 3. GLOBAL_COMMIT -------------->|                            |
    |---- 3. GLOBAL_COMMIT ------------------------------------------->|
    |                                     | [Releases Locks]           | [Releases Locks]
    |<--- 4. ACK -------------------------|                            |
    |<--- 4. ACK ------------------------------------------------------|

The Critical Flaw (Blocking State): If the Coordinator crashes immediately after receiving YES votes but before broadcasting GLOBAL_COMMIT, participants remain locked in the PREPARED state. They cannot release row locks, because they do not know whether the coordinator committed or aborted.

3.2 Saga Patterns: Orchestration vs. Choreography

A Saga is a sequence of local transactions $\{T_1,T_2,\dots ,T_n\}$ with corresponding compensating transactions $\{C_1,C_2,\dots ,C_{n-1}\}$.

• Choreography (Decentralized Events): Each service executes $T_i$, commits locally, and emits a Kafka event. The next service listens to the event and executes $T_{i+1}$. If failure occurs, compensating events are emitted backwards. Problem: Circular event dependencies and difficulty visualizing global workflow state.
• Orchestration (Centralized State Machine): A dedicated orchestrator (e.g. Temporal, AWS Step Functions) coordinates the sequence, explicitly calling RPC endpoints for $T_i$ and invoking $C_i$ if any step fails.

4. Formal Mathematical Derivations

4.1 Latency Cost of 2PC vs. Local Commit

Let network round-trip time between coordinator and participants be $RTT$. Let local disk fsync time for the transaction log be ${\tau }_{\text{fsync}}$.

In a standard single-node transaction: $$T_{\text{local}}={\tau }_{\text{fsync}}$$

In a distributed 2PC transaction across $N$ participants in parallel: $$T_{\text{2PC}}=\underset{\text{Phase 1 (Prepare)}}{\underbrace{RTT+\underset{1\le i\le N}{\max }({\tau }_{\text{prepare},i})}}+\underset{\text{Coordinator Log}}{\underbrace{{\tau }_{\text{coord\_wal}}}}+\underset{\text{Phase 2 (Commit)}}{\underbrace{RTT+\underset{1\le i\le N}{\max }({\tau }_{\text{commit},i})}}$$

Assuming $RTT\approx 2\text{ms}$ and ${\tau }_{\text{fsync}}\approx 1\text{ms}$: $$T_{\text{2PC}}\approx 2+1+1+2+1=7\text{ms}$$ Distributed 2PC introduces a $7\times$ end-to-end latency penalty compared to local $1\text{ms}$ commits while holding contended row locks across participants, capping per-key throughput via Little’s Law:

4.2 Availability Multiplication Penalty: Synchronous 2PC vs. Durable Sagas

In a synchronous 2PC transaction spanning $N$ independent microservice databases and 1 coordinator—each with independent steady-state availability $p\in (0,1)$—every participant and the coordinator must be simultaneously reachable for the transaction to commit:

• Worked Example: Across $N=4$ services (Loyalty, Inventory, Payments, Shipping) plus 1 Coordinator, each boasting $p=99.9\%$ ($0.999$) uptime: $$A_{\text{2PC}}=0.999^5\approx 0.99501(99.501\%\text{ availability}⟹43.7\text{ hours of downtime/year!})$$
• By contrast, an asynchronous Saga persisted in a replicated durable workflow log (Temporal, Apache Kafka) decouples temporal availability: order acceptance requires only the durable orchestrator ($A_{\text{ingest}}\ge 99.99\%$), while transient downstream service restarts are absorbed via exponential backoff retries or clean backward compensation.

5. Edge Cases, Failure Modes & Distributed Anomalies

5.1 The Semantic Anomaly of Sagas: Lack of Isolation

The fundamental compromise of the Saga pattern is that Sagas provide Atomicity, Consistency, and Durability, but NOT Isolation (ACD without I)! Because Step $T_1$ commits immediately to its local database:

• Dirty Reads: Another user can read the intermediate state of $T_1$ before the Saga completes.
• If Step $T_3$ subsequently fails and $C_1$ is executed to rollback $T_1$, the external user has already acted upon data that was retroactively cancelled!

5.2 Out-of-Order Compensations

In an asynchronous network, a compensating transaction $C_i$ might arrive at a service before the forward transaction $T_i$ due to network routing delays! If the service executes $C_i$ (e.g. refund $50) on a non-existent account record, and $T_i$ (charge $50) arrives 2 seconds later, the user keeps the $50 charge and the refund is lost. Remediation: The service must record a “pre-cancellation tombstone” when $C_i$ arrives first, so that when $T_i$ eventually arrives, it is immediately rejected as a no-op.

6. Open-Source Architectural Studies

6.1 CockroachDB Parallel Commits & TiDB Async Commit over Multi-Raft

Distributed SQL databases eliminate the single-node 2PC coordinator bottleneck by storing both transaction status records and row-level lock intents inside Multi-Raft consensus groups:

• CockroachDB Parallel Commits: The transaction coordinator writes provisional write intents to participant ranges and simultaneously writes a STAGING transaction record to the coordinator’s Raft group in parallel. Once both acknowledgments return to the client, the transaction is implicitly committed in $1\times {\text{RTT}}_{\text{Raft}}$, and the STAGING record is asynchronously promoted to COMMITTED.
• TiDB / TiKV Percolator Async Commit & 1PC: TiKV builds on Google Percolator’s primary-lock 2PC model (designating one mutated key as the primary lock and pointing secondary locks to it). In TiDB 5.0+, 1PC commits single-Region transactions in one Raft round without 2PC, while Async Commit records all secondary keys on the primary lock during Prewrite so the client can be acknowledged as soon as all Prewrites succeed—cutting commit latency from $2\times {\text{RTT}}_{\text{Raft}}$ to $1\times {\text{RTT}}_{\text{Raft}}$.

6.2 Temporal: Durable Execution Orchestrator

Temporal executes Sagas as deterministically replayable Go/TypeScript workflows. Every workflow step is recorded in a durable Cassandra/PostgreSQL execution history. If a worker process dies in the middle of executing a compensation, another worker rehydrates the workflow from the event log and resumes compensation from the exact step of failure.

7. Production Incident Post-Mortem (War Stories)

The 2PC Frozen Cluster Disaster

Context: A banking platform used an XA-compliant 2PC distributed transaction manager across Oracle and DB2 databases. The Incident: A high-voltage power surge in the primary data center killed the transaction coordinator server mid-transaction. The Impact: 85 high-volume customer accounts were left in the PREPARED state on the surviving Oracle database. The exclusive table locks blocked all subsequent deposit and withdrawal attempts for those accounts. Because the coordinator server’s local disk was fried, automated recovery could not determine if the transaction had committed or aborted. The Resolution: Database administrators spent 6 hours manually inspecting physical transaction logs and issuing manual COMMIT PREPARED and ROLLBACK PREPARED SQL statements. The Architectural Lesson: Never use blocking 2PC across independent microservices; replace synchronous locks with asynchronous Sagas or TCC resource reservations.

8. Production-Grade TypeScript Simulation: Distributed Saga Orchestrator

This simulation implements a complete state-machine Saga Orchestrator executing multi-step forward transactions with automatic, idempotent backward compensation on failure.

/**
 * Production Saga Orchestration Simulation with Compensating Workflows
 */

export interface SagaStep<TContext> {
  name: string;
  execute: (context: TContext) => Promise<void>;
  compensate: (context: TContext) => Promise<void>;
}

export enum SagaStatus {
  NOT_STARTED = "NOT_STARTED",
  IN_PROGRESS = "IN_PROGRESS",
  COMPLETED = "COMPLETED",
  COMPENSATING = "COMPENSATING",
  COMPENSATED = "COMPENSATED",
  FAILED = "FAILED",
}

export class SagaOrchestrator<TContext> {
  private steps: SagaStep<TContext>[] = [];
  public status: SagaStatus = SagaStatus.NOT_STARTED;
  public executionLog: string[] = [];

  public addStep(step: SagaStep<TContext>): this {
    this.steps.push(step);
    return this;
  }

  public async execute(context: TContext): Promise<boolean> {
    this.status = SagaStatus.IN_PROGRESS;
    const executedSteps: SagaStep<TContext>[] = [];

    for (let i = 0; i < this.steps.length; i++) {
      const step = this.steps[i];
      try {
        this.executionLog.push(`[EXECUTE] Step: ${step.name}`);
        await step.execute(context);
        executedSteps.push(step);
      } catch (err: unknown) {
        const message = err instanceof Error ? err.message : String(err);
        this.executionLog.push(`[FAILURE] Step: ${step.name} failed with: ${message}`);
        this.status = SagaStatus.COMPENSATING;
        await this.rollback(executedSteps, context);
        this.status = SagaStatus.COMPENSATED;
        return false;
      }
    }

    this.status = SagaStatus.COMPLETED;
    this.executionLog.push(`[SUCCESS] Saga completed successfully`);
    return true;
  }

  private async rollback(executedSteps: SagaStep<TContext>[], context: TContext): Promise<void> {
    // Compensate in reverse order
    for (let i = executedSteps.length - 1; i >= 0; i--) {
      const step = executedSteps[i];
      try {
        this.executionLog.push(`[COMPENSATE] Rolling back step: ${step.name}`);
        await step.compensate(context);
      } catch (compErr: unknown) {
        const message = compErr instanceof Error ? compErr.message : String(compErr);
        this.executionLog.push(
          `[CRITICAL] Compensation failed for ${step.name}: ${message}. Requires manual triage!`
        );
      }
    }
  }
}

// Verification Routine
export async function runSagaSimulation() {
  console.log("=== Distributed Saga Orchestration Simulation ===");

  interface OrderContext {
    orderId: string;
    userId: string;
    totalAmount: number;
    inventoryReserved: boolean;
    paymentCharged: boolean;
    shippingScheduled: boolean;
  }

  const context: OrderContext = {
    orderId: "ord_999",
    userId: "usr_42",
    totalAmount: 150,
    inventoryReserved: false,
    paymentCharged: false,
    shippingScheduled: false,
  };

  const saga = new SagaOrchestrator<OrderContext>();

  // Step 1: Inventory
  saga.addStep({
    name: "ReserveInventory",
    execute: async (ctx) => {
      ctx.inventoryReserved = true;
    },
    compensate: async (ctx) => {
      ctx.inventoryReserved = false;
    },
  });

  // Step 2: Payment
  saga.addStep({
    name: "ChargePayment",
    execute: async (ctx) => {
      ctx.paymentCharged = true;
    },
    compensate: async (ctx) => {
      ctx.paymentCharged = false;
    },
  });

  // Step 3: Shipping (Simulate intentional third-party failure)
  saga.addStep({
    name: "ScheduleShipping",
    execute: async () => {
      throw new Error("Carrier address verification rejected: invalid postal code");
    },
    compensate: async (ctx) => {
      ctx.shippingScheduled = false;
    },
  });

  const success = await saga.execute(context);
  console.log(`Saga Succeeded: ${success}`);
  console.log(`Final Saga Status: ${saga.status}`);
  console.log("Execution Trace:");
  for (const log of saga.executionLog) {
    console.log(`  ${log}`);
  }
  console.log("Final Context State:", context);
}

runSagaSimulation();

9. Operational Checklist, Failure Runbooks & Protocol Matrix

Distributed Transaction Comparison Matrix

10. Guided Applied System Design Exercises & Debugging Drills

Drill 1: Designing a Hotel Booking TCC Engine

Design a Try-Confirm-Cancel protocol for an online travel agency reserving rooms across 15 hotel partner APIs.

1. Define the exact state machine transitions for the room entity in the partner’s database (AVAILABLE $\to$ HELD_FOR_TRY $\to$ CONFIRMED).
2. What expiration mechanism ensures that a crashed client does not hold hotel rooms indefinitely?

Drill 2: Resolving Saga Dirty Read Anomalies

A user initiates a flight transfer. The Saga deducts $500 from Account A, and prepares to credit $500 to Account B. In the 2 seconds before Account B is credited, Account A’s mobile app queries its balance and displays the $500 deduction. A sudden network failure causes Account B’s credit to abort, and compensation refunds Account A.

1. How do you design the user interface and database schema (e.g. pending balances) to prevent user panic over transient intermediate states?

11. Summary & Architectural Decision Guide

• Use Consensus-Backed 2PC (CockroachDB Parallel Commits, TiDB/TiKV Percolator, Google Spanner) when all participating shards belong to a single unified distributed SQL cluster where coordinators and participants are replicated Raft/Paxos groups rather than single points of failure.
• Use Orchestrated Sagas (Temporal, AWS Step Functions) across autonomous microservices or third-party HTTP APIs (Stripe, FedEx) where holding cross-service locks is impossible; enforce idempotency keys and pre-cancellation tombstones on every forward and compensating handler.
• Use Try-Confirm-Cancel (TCC) for high-contention financial wallets or inventory reservations where exposing intermediate uncommitted Saga deductions (dirty reads) violates business invariants.

12. Authoritative References & Version Context

• Version & Implementation Baseline: Distributed transaction architectures reflect CockroachDB v23.2 (Parallel Commits), TiDB / TiKV v7.5 (Percolator-inspired 2PC with Async Commit & 1PC), and Temporal Server v1.22+.
• Gray, J. (1978). Notes on Data Base Operating Systems (§5.7 Two-Phase Commit Protocol). Operating Systems: An Advanced Course, Springer LNCS Vol. 60, 393–481.
• Skeen, D. (1981). Nonblocking Commit Protocols. Proceedings of the 1981 ACM SIGMOD International Conference on Management of Data, 133–142.
• Garcia-Molina, H., & Salem, K. (1987). Sagas. Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, 249–259.
• Peng, D., & Dabek, F. (2010). Large-scale Incremental Processing Using Distributed Transactions and Notifications (Google Percolator). Proceedings of USENIX OSDI ‘10, 251–264.
• Taft, R., et al. (2020). CockroachDB: The Resilient Geo-Distributed SQL Database (§4.2 Parallel Commits). Proceedings of ACM SIGMOD ‘20, 1493–1509.