🤖 AI Summary
The surging computational demands of large language model (LLM) training exceed the capacity of homogeneous clusters from a single vendor, necessitating heterogeneous collaborative training across AMD and NVIDIA GPUs. This work proposes a compatibility-oriented hybrid communication architecture that integrates CPU-mediated forwarding and direct device-to-device connectivity, and introduces an innovative CPU-offloaded peer-to-peer transfer mechanism that enables cross-vendor GPU data exchange without host memory staging for the first time. By leveraging multi-NIC parallel transmission and differentiated communication backend scheduling, the system achieves 98% of the throughput of an NVIDIA-only homogeneous cluster on LLaMA-8B and Qwen2-7B models while preserving training stability and correctness.
📝 Abstract
As large language models continue to scale, training demands on compute and system capacity grow rapidly, making single-vendor homogeneous clusters insufficient. This paper presents a technical solution for heterogeneous mixed training in AMD-NVIDIA environments. We first adopt a compatibility-oriented approach based on CPU-Forwarding Communication, with differentiated communication back-end selection across parallel groups and multi-NIC parallel data transfer. To achieve higher performance, we further propose another Device-Direct Communication approach, integrating a CPU-offloading P2P mechanism to enable direct cross-vendor GPU data transfer without host-memory staging. Experiments on LLaMA-8B and Qwen2-7B demonstrate that the proposed Device-Direct Communication approach achieves up to 98% of the throughput of an NVIDIA homogeneous system, while preserving training stability and correctness.