Collective Communication & GPU Interconnect Architecture: Ring All-Reduce, Tree Collectives, and NVLink vs. InfiniBand

1. Problem and Motivation

In modern distributed deep learning, training multi-billion parameter foundation models (Llama 3, GPT-4, Gemini) requires orchestrating clusters consisting of thousands of accelerator chips (such as NVIDIA H100, B200, or AMD MI300X).

While modern tensor accelerators deliver staggering computational power—an NVIDIA H100 SXM5 yields $989.5\text{ TFLOPS}$ dense FP16/BF16 Tensor Core throughput ($1,979\text{ TFLOPS}$ with 2:4 structured sparsity) and $3.35\text{ TB/s}$ HBM3 bandwidth—their real-world distributed training efficiency is fundamentally governed by interconnect communication bandwidth:

+---------------------------------------------------------------------------------------------------+
|                                  THE COMMUNICATION BOTTLENECK                                     |
+---------------------------------------------------------------------------------------------------+
|  1. Massive Gradient Payloads: A 70B parameter model produces 140 GB of FP16 gradients every      |
|     single forward-backward training step.                                                       |
|  2. Centralized Ingestion Collapse: A naive parameter server receiving 140 GB from 512 worker     |
|     GPUs must ingest 71.6 Terabytes of network data every step, saturating any physical switch.  |
|  3. The Physical Interconnect Hierarchy Gap:                                                      |
|     - Intra-Node NVLink 4: 900 GB/sec bidirectional per GPU (sub-microsecond crossbar latency).   |
|     - Inter-Node InfiniBand NDR: 400 Gbps (~50 GB/sec per GPU) (18x bandwidth drop!).            |
|     - Commodity PCIe Gen5: 64 GB/sec (14x slower than NVLink).                                   |
|  4. Synchronization Latency: If GPUs must stall their tensor cores while waiting for network      |
|     gradient exchanges, GPU compute utilization drops below 30%, wasting millions of dollars in   |
|     data center capital expenditure.                                                              |
+---------------------------------------------------------------------------------------------------+

To eliminate central server bottlenecks, modern AI infrastructure relies exclusively on decentralized collective communication algorithms, primarily the Ring All-Reduce and Double-Binary Tree primitives orchestrated by the NVIDIA Collective Communications Library (NCCL).

2. Mental Model & Architectural Overview

Collective communication describes algorithms where a group of $P$ participating processes (GPUs) collectively exchange, reduce, or broadcast data arrays of size $M$ bytes.

SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED AI

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Step 0: Tensor Chunk Partitioning & Phase A: Scatter-Reduce (P - 1 = 3 Steps)

Establishes the ingress and structural boundary across Step 0: Tensor Chunk Partitioning & Phase A: Scatter-Reduce (P - 1 = 3 Steps): Divide tensor of size M into P = 4 equal chunks; GPU i holds: [C0_i, C1_i, C2_i, C3_i]; Chunk size = M / 4 bytes.

Architectural Invariant: Total Bytes = 2 × ((P - 1) / P) × M Independent of P as P grows large (→ 2M).

ARCHITECTURAL FLOW · STEP 2 OF 3

2. Intra-Node: NVLink & NVSwitch & PCIe Gen5 Comparison

Coordinates core data-plane execution and state transitions through Intra-Node: NVLink & NVSwitch & PCIe Gen5 Comparison: NVIDIA H100 / B200 SXM Chassis (8 GPUs); 4th Gen NVLink: 900 GB/sec bidirectional; NVSwitch crossbar: Full all-to-all crossbar.

Architectural Invariant: Latency: < 1 µs, Bandwidth: 900 GB/s.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Hierarchical / 2D Collectives & NCCL Channel Multiplexing

Enforces durability, state convergence, and output verification across Hierarchical / 2D Collectives & NCCL Channel Multiplexing: To hide the 18x network bandwidth gap; 1. Intra-node Reduce-Scatter via NVLink (900 GB/s); 2. Inter-node All-Reduce via InfiniBand (50 GB/s).

Architectural Invariant: Total Step Time = 140ms With Pipelined Bucketing (DDP): Layer N Grad → AllReduce(Layer N) Simultaneous with Layer N-1 Compute Hides 90% of All-Reduce latency!.

The Core Primitives

• All-Reduce: Every GPU starts with an array of size $M$. All arrays are element-wise summed (or averaged), and every GPU finishes with the complete, identical reduced array.
• Reduce-Scatter: Every GPU starts with an array of size $M$. The arrays are element-wise summed, but the final reduced array is partitioned equally among the $P$ GPUs, such that GPU $i$ receives a unique chunk of size $M/P$.
• All-Gather: Every GPU starts with a distinct local chunk of size $M/P$. Every GPU gathers all chunks from all peers, such that all GPUs finish with the full concatenated array of size $M$.
• Broadcast: One root GPU sends an array of size $M$ to all other $P-1$ GPUs.

3. Terminology & Core Mechanics

• NCCL (NVIDIA Collective Communications Library): The de facto standard library providing high-performance collective communication primitives multi-threaded and multi-GPU aware for CUDA.
• NVLink: NVIDIA’s high-speed, direct GPU-to-GPU point-to-point interconnect. On H100 (NVLink 4), each GPU provides 18 links yielding 900 GB/s bidirectional bandwidth.
• NVSwitch: An on-board physical crossbar switch allowing all 8 GPUs within an HGX/DGX chassis to communicate with any other GPU at full 900 GB/s NVLink line rate.
• InfiniBand NDR (Next Data Rate): A high-throughput, low-latency cluster networking standard providing 400 Gbps (50 GB/s) per port with hardware Remote Direct Memory Access (RDMA).
• GPUDirect RDMA: A hardware mechanism that allows network interface cards (NICs) to directly access GPU High-Bandwidth Memory (HBM) over the PCIe/NVLink bus without staging data through host CPU DRAM.
• Hockney Communication Model: An analytical model that decomposes communication time into network startup latency $\alpha$ and per-byte transfer cost $\beta \cdot M$: $$T=\alpha +\beta \cdot M$$

4. Internal Workings & Algorithms: Ring All-Reduce Step-by-Step

In a ring topology, $P$ GPUs are logically arranged in a circular ring: $${\text{GPU}}_0⟶{\text{GPU}}_1⟶{\text{GPU}}_2⟶\cdots ⟶{\text{GPU}}_{P-1}⟶{\text{GPU}}_0$$

Every GPU connects only to its immediate successor (sending data) and immediate predecessor (receiving data). The tensor of size $M$ bytes is divided into $P$ equal chunks: $[C_0,C_1,\dots ,C_{P-1}]$, where each chunk has size: $$\text{Chunk Size}=\frac{M}{P}\text{ bytes}$$

Phase 1: The Scatter-Reduce Phase ($(P-1)$ Steps)

The objective of Scatter-Reduce is to calculate the global sum of each chunk such that each GPU ends up holding the full global sum of exactly one chunk.

Initial State (P = 4 GPUs, 4 Chunks):
GPU 0: [C0_0, C1_0, C2_0, C3_0]
GPU 1: [C0_1, C1_1, C2_1, C3_1]
GPU 2: [C0_2, C1_2, C2_2, C3_2]
GPU 3: [C0_3, C1_3, C2_3, C3_3]

At each step $k\in [0,P-2]$:

1. GPU $i$ sends chunk $(i-k)(\mathrm{mod}P)$ to its successor GPU $(i+1)(\mathrm{mod}P)$.
2. Simultaneously, GPU $i$ receives chunk $(i-k-1)(\mathrm{mod}P)$ from its predecessor GPU $(i-1)(\mathrm{mod}P)$.
3. GPU $i$ performs an in-memory element-wise addition: $$\text{chunk}\leftarrow {\text{chunk}}_{\text{local}}+{\text{chunk}}_{\text{received}}$$

After exactly $P-1$ steps:

• GPU 0 holds the complete global sum: ${\sum }_{j=0}^{P-1}C{1}_j$
• GPU 1 holds the complete global sum: ${\sum }_{j=0}^{P-1}C{2}_j$
• GPU 2 holds the complete global sum: ${\sum }_{j=0}^{P-1}C{3}_j$
• GPU 3 holds the complete global sum: ${\sum }_{j=0}^{P-1}C{0}_j$

Phase 2: The All-Gather Phase ($(P-1)$ Steps)

Now that each GPU holds the complete sum of one chunk, that sum must be shared with all other $P-1$ GPUs.

At each step $k\in [0,P-2]$:

1. GPU $i$ sends its fully reduced chunk to its successor.
2. GPU $i$ receives the fully reduced chunk from its predecessor and overwrites its local partial chunk (no summation required).
3. The chunk travels one hop around the ring per step.

After exactly $P-1$ steps: All $P$ GPUs hold the identical, fully reduced chunks $[C_0,C_1,C_2,C_3]$!

5. Mathematical Derivations & Performance Model

Let us mathematically prove the bandwidth optimality and asymptotic invariance of Ring All-Reduce.

1. Data Volume Transmitted per GPU

In each step of the Scatter-Reduce phase:

• Each GPU sends a single chunk of size $\frac{M}{P}$ bytes.
• Number of steps = $P-1$. $${\text{Volume}}_{\text{Scatter-Reduce}}=(P-1)\times \frac{M}{P}=\frac{P-1}{P}\cdot M$$

In each step of the All-Gather phase:

• Each GPU sends a single chunk of size $\frac{M}{P}$ bytes.
• Number of steps = $P-1$. $${\text{Volume}}_{\text{All-Gather}}=(P-1)\times \frac{M}{P}=\frac{P-1}{P}\cdot M$$

Adding both phases together: $$\text{Total Transmitted per GPU}=2\times \frac{P-1}{P}\cdot M$$

The Asymptotic Limit:

2. Time Complexity under the Hockney Model

Let:

• $\alpha$: Latency of initiating a network transmission (hop latency in seconds)
• $\beta =\frac{1}{B}$: Per-byte transfer time, where $B$ is link bandwidth (bytes/sec)
• $\gamma$: Computational time to perform element-wise floating-point addition per byte

The total time for Ring All-Reduce is: $$T_{\text{Ring}}=2(P-1)\alpha +2\cdot \frac{P-1}{P}\cdot M\cdot \beta +\frac{P-1}{P}\cdot M\cdot \gamma$$

For large payloads $M$ (e.g., hundreds of megabytes of gradients), the bandwidth term $\frac{2(P-1)}{P}M\beta \approx 2M\beta$ completely dominates the latency term $2(P-1)\alpha$.

3. Design Alternatives & Collective Comparison Matrix: Ring vs. Tree vs. NVLS

+---------------------------------------------------------------------------------------------------+
|                                 RING vs. TREE ALL-REDUCE COMPARISON                               |
+---------------------------------------------------------------------------------------------------+
| Metric                 | Ring All-Reduce                        | Double-Binary Tree All-Reduce   |
+------------------------+----------------------------------------+---------------------------------+
| Network Latency Steps  | 2(P - 1) (Linear in P)                 | ~ 2 * log2(P) (Logarithmic in P)|
| Bandwidth Factor       | 2 * (P - 1) / P (Optimal: -> 2M)       | ~ 2M (two complementary trees)  |
| Optimal Payload Size   | Large tensors (> 100 MB)               | Small/Medium tensors (< 10 MB)  |
| Straggler Sensitivity  | High: 1 slow link stalls entire ring   | Medium: Isolated to sub-tree    |
+------------------------+----------------------------------------+---------------------------------+

When $P=1,024$ GPUs:

• Ring latency requires $2\times (1,024-1)=2,046\text{ hops}$.
• Balanced binary-tree reduction + broadcast requires $2\times {\log }_2(1,024)=20\text{ depth levels}$ ($\approx 100\times$ fewer serial startup latencies!).

In NCCL 2.20+ on Hopper/Blackwell systems, a third alternative—NVLink SHARP (NVLS) / InfiniBand SHARP—delegates floating-point summation directly into the NVSwitch3 / Quantum-2 switch silicon, cutting intra-node traffic from $2\frac{P-1}{P}M$ down to $1\cdot M$ read + $1\cdot M$ multicast write with $O(1)$ switch hops. NCCL dynamically selects Tree, Ring, or NVLS/CollNet per collective based on payload bytes $M$, communicator size $P$, and detected fabric topology.

6. Hardware Interconnect Realities: NVLink vs. InfiniBand

A critical error in naive distributed systems models is treating all GPU-to-GPU links as uniform. In reality, modern AI clusters feature a two-tier non-uniform bandwidth hierarchy:

[ DGX / HGX Chassis: 8 x H100 GPUs ]
+-------------------------------------------------------------------------+
| GPU 0 <======== NVLink 4 Crossbar (900 GB/sec) ========> GPU 1 ... GPU 7|
+-------------------------------------------------------------------------+
       |             |             |             |             |
     PCIe          PCIe          PCIe          PCIe          PCIe
       v             v             v             v             v
  [CX7 NIC 0]   [CX7 NIC 1]   [CX7 NIC 2]   [CX7 NIC 3]   [CX7 NIC 4] ...
   400 Gbps      400 Gbps      400 Gbps      400 Gbps      400 Gbps
  (50 GB/s)     (50 GB/s)     (50 GB/s)     (50 GB/s)     (50 GB/s)
       \             \             |             /             /
+-------------------------------------------------------------------------+
|            InfiniBand Quantum-2 NDR Inter-Node Leaf/Spine Fabric        |
+-------------------------------------------------------------------------+

The 18x Bandwidth Gap:

• Intra-Chassis (NVLink 4 + NVSwitch): $900\text{ GB/sec}$ bidirectional bandwidth per GPU. Latency $<0.8\mu \text{s}$.
• Inter-Chassis (InfiniBand NDR 400G): $400\text{ Gbps}=50\text{ GB/sec}$ per GPU. Latency $\approx 2.5\mu \text{s}$.

Hierarchical 2D All-Reduce

To avoid running a slow global ring through 50 GB/s InfiniBand links for every step, NCCL employs Hierarchical All-Reduce:

1. Intra-Node Reduce-Scatter: The 8 GPUs within each node reduce their chunks using the blisteringly fast 900 GB/s NVLink bus.
2. Inter-Node All-Reduce: Only 1 reduced chunk per GPU is sent over the 50 GB/s InfiniBand network to peer GPUs on other chassis.
3. Intra-Node All-Gather: The fully reduced chunks are broadcast locally inside each node over NVLink.

7. Production Code Engine: Ring All-Reduce Simulator

The following TypeScript implementation provides a working simulation of the Ring All-Reduce algorithm, verifying numerical correctness, chunk splitting, scatter-reduce accumulation, and all-gather distribution:

/**
 * Production-Grade Ring All-Reduce Simulator
 * Demonstrates Scatter-Reduce and All-Gather phases with verification.
 */

export class GPUWorker {
  public data: Float64Array;

  constructor(
    public readonly id: number,
    initialValues: number[]
  ) {
    this.data = new Float64Array(initialValues);
  }

  public getChunk(chunkIndex: number, chunkSize: number): Float64Array {
    const start = chunkIndex * chunkSize;
    return this.data.slice(start, start + chunkSize);
  }

  public setChunk(chunkIndex: number, chunkSize: number, source: Float64Array): void {
    const start = chunkIndex * chunkSize;
    this.data.set(source, start);
  }

  public reduceChunk(chunkIndex: number, chunkSize: number, received: Float64Array): void {
    const start = chunkIndex * chunkSize;
    for (let i = 0; i < chunkSize; i++) {
      this.data[start + i] += received[i];
    }
  }
}

export class RingAllReducer {
  private readonly numGPUs: number;
  private readonly tensorSize: number;
  private readonly chunkSize: number;

  constructor(public readonly gpus: GPUWorker[]) {
    this.numGPUs = gpus.length;
    this.tensorSize = gpus[0].data.length;

    if (this.tensorSize % this.numGPUs !== 0) {
      throw new Error(`Tensor size ${this.tensorSize} must be divisible by GPU count ${this.numGPUs}`);
    }
    this.chunkSize = this.tensorSize / this.numGPUs;
  }

  /**
   * Executes full Ring All-Reduce: Scatter-Reduce followed by All-Gather.
   */
  public execute(): { totalTransferredPerGPU: number; stepsExecuted: number } {
    const P = this.numGPUs;
    const M = this.tensorSize;
    let transferredBytes = 0;

    // ==========================================
    // PHASE 1: SCATTER-REDUCE (P - 1 steps)
    // ==========================================
    for (let step = 0; step < P - 1; step++) {
      // Temporary buffers to simulate parallel synchronous ring transfers
      const sendBuffers: Float64Array[] = [];

      for (let i = 0; i < P; i++) {
        // GPU i sends chunk: (i - step + P) % P
        const sendChunkIdx = (i - step + P) % P;
        sendBuffers[i] = this.gpus[i].getChunk(sendChunkIdx, this.chunkSize);
      }

      for (let i = 0; i < P; i++) {
        // GPU i receives from predecessor GPU (i - 1 + P) % P
        const recvChunkIdx = (i - step - 1 + P) % P;
        const predecessorId = (i - 1 + P) % P;
        const receivedData = sendBuffers[predecessorId];

        // Sum received data into local chunk
        this.gpus[i].reduceChunk(recvChunkIdx, this.chunkSize, receivedData);
      }

      transferredBytes += this.chunkSize * 8; // Float64 = 8 bytes
    }

    // ==========================================
    // PHASE 2: ALL-GATHER (P - 1 steps)
    // ==========================================
    for (let step = 0; step < P - 1; step++) {
      const sendBuffers: Float64Array[] = [];

      for (let i = 0; i < P; i++) {
        // GPU i sends its fully reduced chunk: (i - step + 1 + P) % P
        const sendChunkIdx = (i - step + 1 + P) % P;
        sendBuffers[i] = this.gpus[i].getChunk(sendChunkIdx, this.chunkSize);
      }

      for (let i = 0; i < P; i++) {
        // GPU i receives and overwrites local chunk
        const recvChunkIdx = (i - step + P) % P;
        const predecessorId = (i - 1 + P) % P;
        const receivedData = sendBuffers[predecessorId];

        this.gpus[i].setChunk(recvChunkIdx, this.chunkSize, receivedData);
      }

      transferredBytes += this.chunkSize * 8;
    }

    return {
      totalTransferredPerGPU: transferredBytes,
      stepsExecuted: 2 * (P - 1),
    };
  }
}

function runDemo(): void {
  console.log("=== GPU Interconnect: Ring All-Reduce Simulation ===");
  const numGPUs = 4;
  const tensorSize = 8;
  const gpus = Array.from({ length: numGPUs }, (_, id) =>
    new GPUWorker(id, Array.from({ length: tensorSize }, (_, i) => (id + 1) * 10 + i))
  );
  console.log(`Cluster: ${numGPUs} GPUs, Tensor size: ${tensorSize} elements (${tensorSize * 8} bytes)`);
  console.log(`GPU 0 Initial: [${Array.from(gpus[0].data).join(", ")}]`);
  console.log(`GPU 3 Initial: [${Array.from(gpus[3].data).join(", ")}]`);
  const reducer = new RingAllReducer(gpus);
  const result = reducer.execute();
  console.log(`Ring All-Reduce executed: ${result.stepsExecuted} communication steps`);
  console.log(`Transferred per GPU: ${result.totalTransferredPerGPU} bytes`);
  console.log(`GPU 0 Synchronized: [${Array.from(gpus[0].data).join(", ")}]`);
  console.log(`GPU 3 Synchronized: [${Array.from(gpus[3].data).join(", ")}]`);
  const match = Array.from(gpus[0].data).every((val, idx) => val === gpus[3].data[idx]);
  console.log(`All GPUs bitwise synchronized: ${match}`);
}

runDemo();

8. Compute-Communication Overlapping & Gradient Bucketing

In a synchronous distributed training step, executing backpropagation sequentially followed by collective communication leaves the GPU cores completely idle during network transfers:

Sequential Execution (No Overlap):
[ Forward Pass (60ms) ] -> [ Backward Pass (120ms) ] -> [ All-Reduce (45ms) ] -> [ Optimizer (15ms) ]
Total Step Time = 240ms (GPU Idle for 45ms!)

Pipelined Overlapping in PyTorch DDP

PyTorch DDP overcomes this through Gradient Bucketing:

Overlapped Execution (Pipelined Bucketing):
Layer 4 Compute (30ms) ---> Layer 4 Gradients Ready
                                |
Layer 3 Compute (30ms)          +---> NCCL Async All-Reduce(Bucket 2) [CUDA Stream 1]
                                |     (Transfers concurrently over InfiniBand!)
Layer 2 Compute (30ms)          +---> NCCL Async All-Reduce(Bucket 1) [CUDA Stream 1]
                                |
Layer 1 Compute (30ms)          v
All-Reduce finishes almost simultaneously with the final layer computation!
Total Step Time = 198ms (Hides ~90% of All-Reduce latency!)

1. Bucket Partitioning: Model parameters are assigned to contiguous memory buckets (default 25 MB).
2. Reverse Autograd Hook: During backpropagation, as soon as all parameters in a bucket receive their gradients, DDP immediately schedules an asynchronous non-blocking ncclAllReduce call on a dedicated communication CUDA stream.
3. Stream Synchronization: The primary compute stream only synchronizes with the communication stream at the very end of the backward pass, just before the optimizer step.

9. Failure Modes, Network Stragglers & Diagnostics

10. Knowledge Check

Review the core collective communication principles, then complete the 4-question interactive assessment quiz at the end of this lesson:

1. Parameter Server Bottleneck: Articulate why centralized parameter servers scale as $O(P\cdot M)$, while Ring All-Reduce scales as $O(M)$.
2. Ring Volume Derivation: Prove step-by-step why the total data transferred per GPU in Ring All-Reduce equals $2\cdot \frac{P-1}{P}\cdot M$.
3. Double-Binary Trees: Explain why Tree collectives provide $O({\log }_{2}P)$ latency steps and when they should be favored over Ring collectives.
4. Compute-Communication Overlap: Describe how PyTorch DDP uses gradient bucketing and CUDA streams to hide network latency during backpropagation.

11. Practical Exercises & Deliverables

Exercise 1: Implement Bidirectional Ring All-Reduce

Problem: The single ring implementation in Section 7 transfers data only in the clockwise direction, utilizing only 50% of full-duplex network links.
Deliverable:

1. Extend the RingAllReducer TypeScript class to maintain two concurrent rings: Ring 0 (clockwise) and Ring 1 (counter-clockwise).
2. Divide the tensor into $2P$ chunks, routing half through Ring 0 and half through Ring 1 simultaneously.
3. Verify that both rings complete without data races and that all GPU tensors match the exact ground truth sum.

Exercise 2: Benchmark NCCL All-Reduce Bus Bandwidth

Problem: Measure the real-world bus bandwidth of a multi-GPU system and compare it to theoretical line rate.
Deliverable:

1. Run nccl-tests (all_reduce_perf -b 8M -e 1G -f 2 -g 8) on an 8-GPU node.
2. Record the reported Algorithm Bandwidth ($\text{algbw}=S/t$) and Bus Bandwidth ($\text{busbw}=2\cdot \frac{P-1}{P}\cdot \frac{S}{t}$), where $S$ is tensor size in bytes and $t$ is collective wall-clock time.
3. Plot achieved bus bandwidth against message size from 8 MB to 1 GB. Identify the inflection point where bus saturation transitions from latency-bound ($2(P-1)\alpha$ dominant) to bandwidth-bound ($2\frac{P-1}{P}S\beta$ dominant).

export function executeBidirectionalRingAllReduce(gpus: GPUWorker[]): void {
  const P = gpus.length;
  const subChunkSize = gpus[0].data.length / (2 * P);

  // Scatter-Reduce (P - 1 steps): Ring 0 clockwise on [0..P-1], Ring 1 counter-clockwise on [P..2P-1]
  for (let step = 0; step < P - 1; step++) {
    const cwSend = gpus.map((g, i) => g.getChunk((i - step + P) % P, subChunkSize));
    const ccwSend = gpus.map((g, i) => g.getChunk(P + ((i + step) % P), subChunkSize));

    for (let i = 0; i < P; i++) {
      const cwRecvChunk = (i - step - 1 + P) % P;
      const ccwRecvChunk = P + ((i + step + 1) % P);
      gpus[i].reduceChunk(cwRecvChunk, subChunkSize, cwSend[(i - 1 + P) % P]);
      gpus[i].reduceChunk(ccwRecvChunk, subChunkSize, ccwSend[(i + 1) % P]);
    }
  }

  // All-Gather (P - 1 steps)
  for (let step = 0; step < P - 1; step++) {
    const cwSend = gpus.map((g, i) => g.getChunk((i - step + 1 + P) % P, subChunkSize));
    const ccwSend = gpus.map((g, i) => g.getChunk(P + ((i + step - 1 + P) % P), subChunkSize));

    for (let i = 0; i < P; i++) {
      const cwRecvChunk = (i - step + P) % P;
      const ccwRecvChunk = P + ((i + step) % P);
      gpus[i].setChunk(cwRecvChunk, subChunkSize, cwSend[(i - 1 + P) % P]);
      gpus[i].setChunk(ccwRecvChunk, subChunkSize, ccwSend[(i + 1) % P]);
    }
  }
}

12. Summary & Architectural Decision Guide

13. Authoritative References & Version Context

• Version & Hardware Baseline: Verified against NVIDIA NCCL 2.20+, nccl-tests v2.13+, PyTorch 2.3+ (DistributedDataParallel & ProcessGroupNCCL), NVIDIA H100 SXM5 (NVLink 4.0 @ 900 GB/s bidirectional, NVSwitch3), and NVIDIA Quantum-2 InfiniBand NDR (400 Gbps / 50 GB/s per port).
• Patarasuk & Yuan (JPDC 2009) — Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations: Formal proof that Ring All-Reduce achieves the theoretical lower bound $2\frac{P-1}{P}M$ on homogeneous ring networks.
• Sanders, Speck & Träff (Parallel Computing 2009) — Two-Tree Algorithms for Full Bandwidth Broadcast, Reduction and Scan: Mathematical foundation for NCCL’s complementary Double-Binary Tree collectives.
• Thakur, Rabenseifner & Gropp (IJHPCA 2005) — Optimization of Collective Communication Operations in MPICH: Hockney $\alpha \text{–}\beta \text{–}\gamma$ complexity analysis comparing Ring, Recursive Halving-Doubling, and Rabenseifner algorithms.
• Li et al. (VLDB ‘20) — PyTorch Distributed: Experiences on Accelerating Data Parallel Training: Architecture of reverse-order gradient bucketing and compute-communication stream overlapping in PyTorch DDP.
• NVIDIA NCCL Documentation — NVIDIA Collective Communications Library (NCCL) User Guide & Bus Bandwidth Formula and NVIDIA/nccl-tests Performance Specification.