Unawesome AI Fabric Engineering

Networking that moves data between GPUs in AI clusters: RDMA, GPU-to-NIC data paths, collectives, transports, and cluster fabrics.

Written for GPU performance engineers moving into the network.

It assumes you already know GPU kernels, profiling, inference engines, and the basics of distributed inference. The list is ordered from one NIC to one GPU-NIC path, collectives, inference transfer, transports, and whole fabrics. Read Start here first. After that, use it as a reference.

Every resource assumes real NICs, switches, and GPUs. See Footnotes.

Contents

• Start here: the minimum mental model
• 1. RDMA fundamentals
  • Verbs and memory registration
  • NIC behavior
  • Measurement
• 2. The GPU-to-NIC data path
  • GPUDirect and PCIe
  • GPU-initiated networking
• 3. Collectives
  • Algorithms
  • Libraries
• 4. Point-to-point transfer for inference
• 5. Transport and congestion control
• 6. Fabric design for AI clusters
• 7. Scale-up fabrics
• 8. Cloud fabrics
• 9. Operating fabrics at scale
• Frontier

Start here: the minimum mental model

Read these in order.

• How to Think About GPUs - NVLink, NVSwitch, scale-out InfiniBand, and collective costs from the GPU's point of view.
• RDMA Aware Networks Programming User Manual - Verbs objects, queue pairs, memory registration, and transport types on NVIDIA NICs.
• Design Guidelines for High Performance RDMA Systems - How NIC caches, PCIe transactions, and verb choice decide RDMA throughput.
• GPUDirect RDMA - How a PCIe device reads and writes GPU memory directly, and what the driver must pin.
• RDMA over Commodity Ethernet at Scale - RoCEv2 with PFC in a production datacenter, including deadlocks and pause storms.
• RDMA over Ethernet for Distributed Training at Meta Scale - A RoCE fabric built for AI training: topology, routing, and congestion choices.
• Demystifying NCCL - NCCL's protocols, channels, and ring and tree algorithms, measured.
• NCCL performance methodology - Algorithm bandwidth versus bus bandwidth, and how to compare a collective against link speed.

1. RDMA fundamentals

Verbs and memory registration

• rdma-core - The userspace verbs library, NIC providers, and man pages.
• ibv_reg_mr(3) - Memory regions, access flags, on-demand paging, and dma-buf registration.
• ibv_advise_mr(3) - Prefetching on-demand paging translations before the hot path.
• Page Fault Support for Network Controllers - The design behind on-demand paging in RDMA NICs.
• Linux InfiniBand subsystem - Kernel-side user verbs, MAD access, and core interfaces.

NIC behavior

• Mellanox Adapters Programmer's Reference Manual - The ConnectX command interface, work queue formats, and offloads that later ConnectX parts extend.
• FaSST - Two-sided datagram RPCs over RDMA, and when they beat one-sided reads.
• eRPC: Datacenter RPCs can be General and Fast - Fast RPCs over lossy fabrics with congestion control in software.
• Collie - A systematic search for RDMA NIC performance anomalies.
• Husky: Understanding RDMA Microarchitecture Resources for Performance Isolation - Shared NIC resources that let one tenant degrade another.

Measurement

• perftest - ib_write_bw, ib_read_lat, and the standard verbs benchmarks.
• nccl-tests - The standard collective benchmarks.
• InfiniBand Architecture Specification - How to obtain the specification that defines InfiniBand and RoCE transport.

2. The GPU-to-NIC data path

GPUDirect and PCIe

• Linux dma-buf - The kernel buffer-sharing mechanism used to register GPU memory without a peer-memory module.
• PCI peer-to-peer DMA - Kernel support and constraints for device-to-device DMA across PCIe.
• NCCL troubleshooting - PCI ACS, IOMMU, and platform settings that silently break or slow GPU-NIC paths.
• NCCL environment variables - Topology files, NIC selection, GPUDirect levels, and InfiniBand settings.

GPU-initiated networking

• NVSHMEM and GPUDirect Async - InfiniBand GPUDirect Async: GPU threads post RDMA work to the NIC directly.
• NVSHMEM - The GPU-initiated communication library behind it.
• DeepEP - Expert dispatch and combine over NVLink and RDMA, including GPU-initiated low-latency kernels.
• DOCA GPUNetIO - GPU-controlled packet I/O on ConnectX NICs.
• Device Memory TCP - TCP that receives payloads directly into device memory.

3. Collectives

Algorithms

• Optimization of Collective Communication Operations in MPICH - Ring, recursive doubling, and Rabenseifner algorithms with their cost models.
• Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations - Why ring all-reduce reaches the bandwidth lower bound.
• TACCL - Synthesizing topology-aware collective algorithms.
• Swing - All-reduce on torus fabrics by short-cutting rings.
• SHArP - In-network reduction inside InfiniBand switches.

Libraries

• NCCL documentation - The reference for NVIDIA collectives.
• NCCL net plugins - The interface a transport implements to carry NCCL traffic.
• NCCL 2.23 scaling algorithm - Parallel Aggregated Trees for all-gather and reduce-scatter with small and medium messages at scale.
• Collective Communication for 100k+ GPUs - Meta's collective stack at 100k-GPU scale.
• MSCCL++ and repository - GPU-driven communication primitives for inference.
• RCCL - AMD's NCCL-compatible collectives.
• UCX - The transport framework under many MPI and inference-transfer backends.

4. Point-to-point transfer for inference

• NIXL - Transfers inference state across GPU, CPU, storage, and network backends.
• Mooncake - The RDMA transfer engine and KV store behind Kimi serving.
• fabric-lib: RDMA Point-to-Point Communication for LLM Systems - One-sided writes with completion counters at 400 Gbps on both ConnectX-7 and EFA, used for KV transfer, RL weight updates, and MoE routing.
• pplx-kernels - Expert-parallel all-to-all kernels.
• UCCL - Collectives, point-to-point, and expert parallelism across NIC vendors.
• Insights into DeepSeek-V3 - Multi-plane fat trees and network-hardware co-design for MoE.
• PD disaggregation and large-scale EP on 96 H100 GPUs - A measured multi-node deployment of DeepSeek with KV transfer and expert parallelism.
• llm-d PD disaggregation guide - A deployable prefill/decode split with RDMA KV transfer.
• llm-d on AMD Instinct over RoCE - Prefill/decode disaggregation tuned on MI300X bare metal over RoCEv2.

5. Transport and congestion control

• DCQCN - ECN-based congestion control for RoCEv2.
• TIMELY - RTT-based congestion control in the datacenter.
• HPCC - Congestion control from in-network telemetry.
• Swift - Delay-based congestion control in production.
• Revisiting Network Support for RDMA - RDMA without a lossless fabric.
• SRD - AWS's multipath, out-of-order reliable datagram transport behind EFA.
• Falcon - Google's hardware-offloaded reliable transport, published through OCP.
• Ultra Ethernet 1.0.3 Specification - The scale-out transport specification.

6. Fabric design for AI clusters

• Building Meta's GenAI Infrastructure - Two 24k-GPU clusters, one on RoCE and one on InfiniBand.
• Alibaba HPN - A dual-plane, two-tier network for LLM training.
• Rail-only - Dropping the spine when traffic stays within rails.
• Fire-Flyer AI-HPC - A cost-driven fabric and all-reduce co-design.
• Jupiter Evolving - Optical circuit switches in a production datacenter network.
• DGX SuperPOD reference architecture - Compute, storage, and management fabrics for an H100 SuperPOD.

7. Scale-up fabrics

• Multi-node NVLink tuning guide - NVLink domains and their InfiniBand boundary in GB200 NVL systems.
• MI300X acceptance guide - Node-level validation for AMD Instinct MI300X systems.
• UALink 1.0 Specification - An open scale-up interconnect.

8. Cloud fabrics

• Elastic Fabric Adapter - AWS's OS-bypass network for AI and HPC.
• aws-ofi-nccl - The NCCL plugin that carries collectives over libfabric and EFA.
• libfabric - The Open Fabric Interfaces library and its providers.
• Azure InfiniBand setup - InfiniBand on Azure HPC and GPU VMs.
• GPUDirect on Google Cloud A3 - GPUDirect-TCPX and TCPXO on A3 instances.
• OCI cluster networks - RDMA cluster networking for bare-metal GPU shapes.

9. Operating fabrics at scale

• MegaScale - Training on more than 10,000 GPUs, with network diagnosis and fault tolerance.
• The Llama 3 Herd of Models - The infrastructure section: network, collectives, and interruptions at 16k GPUs.
• C4 - Real-time detection of communication anomalies in large training jobs.
• SuperBench - Proactive validation of cloud AI infrastructure.
• Understanding Stragglers in Large Model Training - What-if analysis of straggler causes from production traces.

Frontier

Verified on 2026-10-05. Kept separate from the core list because the evidence changes quickly.

Watchlist

• Ultra Ethernet NICs and switches in production, pending measured deployments.
• Adaptive routing and packet spraying on Ethernet AI fabrics, pending reproducible measurements.
• Co-packaged optics switches in deployed GPU clusters.
• UALink and Ethernet-based scale-up silicon, pending shipped systems.
• GPU-initiated networking on non-NVIDIA NICs.
• KV-cache transfer across heterogeneous GPU fleets with measured tail latency.

Footnotes

Hardware policy

Every resource assumes real NICs, switches, and GPUs. Software RDMA emulation and network simulators are excluded: their numbers do not transfer, and they cannot exercise GPUDirect, NIC offloads, congestion control, or adaptive routing.

Source policy

A core source must be one of the following:

• the paper that introduced the mechanism;
• the specification or official documentation that defines it;
• the repository that implements it;
• an operator or implementer report with hardware, measurements, and enough detail to reproduce the result.

Performance claims need the NIC, switch, topology, message sizes, software versions, and baseline. Otherwise the number is omitted.

Contributing

See CONTRIBUTING.md before proposing a resource. python3 scripts/check_links.py checks every link.