RDMA Explained: How to Achieve Microsecond-Level Network Latency

1. What Is RDMA, and Why Is It Non-Negotiable for AI Training

RDMA (Remote Direct Memory Access) lets a NIC read and write remote-server memory directly — bypassing the OS, bypassing the CPU, bypassing the Socket protocol stack.

On a conventional TCP/IP network, an 8 KB packet traveling from a sender application to a receiver application must pass through:

1. Sender app → copy into kernel Socket Buffer;
2. Kernel → TCP/IP stack → copy into NIC DMA Buffer;
3. Network transmission;
4. Receiver NIC → copy into kernel → protocol stack → user space.

Four memory copies + two context switches + interrupt handling — the software path alone burns 50-100μs of latency and 20-40% of CPU. For AI training that synchronizes gradients every 100ms, this is a massive waste.

2. Core Value of RDMA

• Ultra-low latency: down to 1-2 μs, roughly 50x faster than TCP/IP;
• Zero CPU usage: data flows directly between NIC and application memory, CPU stays idle;
• Near-line-rate bandwidth: 10G/25G/100G NICs reach theoretical maximum;
• Hardware flow control: PFC + ECN deliver lossless forwarding, no drops, no retransmit.

3. Three Mainstream RDMA Protocols

3.1 RoCEv2 (RDMA over Converged Ethernet v2)

UDP-based lossless Ethernet RDMA — the enterprise mainstream today:

• Runs over standard Ethernet, compatible with existing infrastructure;
• Switches must support DCB (Data Center Bridging) + PFC + ECN;
• Widely supported by Huawei, H3C, Ruijie, Cisco mainstream switches;
• Typical latency: 3-5 μs.

3.2 InfiniBand

Dedicated high-performance protocol championed by NVIDIA:

• Hardware flow control — natively lossless, no extra switch configuration;
• Lowest latency (0.6-1 μs), highest bandwidth (NDR 400G / XDR 800G);
• Requires dedicated IB switches (Mellanox/NVIDIA Quantum);
• Closed ecosystem, expensive, but ultimate performance.

3.3 iWARP (Internet Wide Area RDMA Protocol)

TCP-based RDMA; best routability but weaker performance than RoCEv2; market share declining.

4. Protocol Comparison

5. RDMA in AI Training Scenarios

5.1 GPUDirect RDMA

NVIDIA GPUs can access each other's VRAM directly via RDMA NICs, bypassing host memory and CPU. This is the core acceleration for 10K-card training — GPU-to-GPU latency drops 80%, bandwidth rises 3-5x.

5.2 AllReduce Collective Communication

90% of large-model training performance depends on the network:

• RDMA keeps every gradient-sync P99 under 20μs;
• NCCL / Gloo collective libraries fully support RDMA — no app code changes needed;
• SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs in-switch reduction on InfiniBand, saving another 50% of communication time.

5.3 Parameter Server (PS) Architecture

In PS architectures, frequent parameter pulls and gradient pushes between workers and servers benefit from RDMA one-sided operations — CPU overhead drops under 5%.

6. Key Deployment Points

6.1 Lossless Network Configuration

# Switch-side PFC (Huawei example)

priority-flow-control mode auto

priority-flow-control priority 3 enable



# Configure ECN

qos wred queue 3 low-limit 20 high-limit 80 discard-probability 10

6.2 Enable RoCEv2 on Server Side

# Load rdma_rxe / mlx5_ib kernel modules

modprobe rdma_rxe

rdma link add rxe_0 type rxe netdev eth0

# Verify RDMA devices

ibv_devices

ibv_devinfo

6.3 Performance Test

# Server side

ib_write_bw -d rxe_0 --report_gbits

# Client side

ib_write_bw -d rxe_0 <server_ip> --report_gbits

# Expected: near 10Gbps / 25Gbps / 100Gbps line rate

7. EZMAX NIC in AI Fabric Design

8. Common Pitfalls and Diagnostics

• Pitfall 1: RoCEv2 comes up but latency stays high — switch DCB not enabled; recheck PFC config;
• Pitfall 2: Single-hop OK, cross-rack drops packets — ECN thresholds need fine-tuning;
• Pitfall 3: GPUDirect RDMA not working — needs IOMMU passthrough + NVIDIA peermem kernel module;
• Pitfall 4: 10K-cluster performance collapse — PFC deadlock, adjust buffer to dynamic allocation.

Conclusion: RDMA has moved from HPC to mainstream data centers — it is the de facto standard for AI training, distributed storage, and low-latency financial trading. EZMAX NETI710 10G NICs fully support RoCEv2 and are the value-optimized choice for building enterprise-grade RDMA fabrics. Combined with correctly configured DCB/PFC/ECN switches, you can build a microsecond-latency lossless network.