Remote Direct Memory Access (RDMA) communication is critical for emerging cross-data-center applications. Particularly in distributed training of large AI models requiring real-time synchronization of massive parameters, as well as scientific computing and meteorological prediction fields that rely on low-latency data exchange, RDMA has become a key infrastructure supporting their core performance. However, existing RDMA technologies are constrained by critical bottlenecks such as protocol stack incompatibility, insufficient congestion control in high-latency environments, and sensitivity to packet loss, severely limiting their large-scale deployment. This paper proposes IB-APC, which extends high-performance lossless RDMA to long-distance links through a co-design of software and hardware. This technology ensures high throughput over long-distance links while achieving interoperability between InfiniBand and Ethernet. It also supports remote RDMA functionality, requiring only minimal additional switches to be added to existing intra-data-center networks. Evaluation results demonstrate that over a 500 km distance, compared to native RDMA, IB-APC achieves a 107-fold increase in bandwidth and reduces the average FCT by up to 96.2%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

IB-APC: InfiniBand-Ethernet Protocol Conversion for Scalable Long-Distance RDMA Communication

  • Anbang Li,
  • Meng Guo,
  • Alexander Arkhipov,
  • Anjun Liu,
  • Ting Xia,
  • Jiaxiang Wang,
  • Yixin Qin,
  • Yunbo Yan,
  • Xiaojuan Meng

摘要

Remote Direct Memory Access (RDMA) communication is critical for emerging cross-data-center applications. Particularly in distributed training of large AI models requiring real-time synchronization of massive parameters, as well as scientific computing and meteorological prediction fields that rely on low-latency data exchange, RDMA has become a key infrastructure supporting their core performance. However, existing RDMA technologies are constrained by critical bottlenecks such as protocol stack incompatibility, insufficient congestion control in high-latency environments, and sensitivity to packet loss, severely limiting their large-scale deployment. This paper proposes IB-APC, which extends high-performance lossless RDMA to long-distance links through a co-design of software and hardware. This technology ensures high throughput over long-distance links while achieving interoperability between InfiniBand and Ethernet. It also supports remote RDMA functionality, requiring only minimal additional switches to be added to existing intra-data-center networks. Evaluation results demonstrate that over a 500 km distance, compared to native RDMA, IB-APC achieves a 107-fold increase in bandwidth and reduces the average FCT by up to 96.2%.