distributed training

This work addresses the challenges of low communication efficiency and system complexity in large-scale AI training across geographically distributed data centers. It presents the first systematic characterization and joint optimization of three critical dimensions in “scale-across” training: parallelism strategy deployment, job scheduling, and network transport. By co-designing parallel placement, scheduling policies, and advanced networking techniques—and validating the approach through both real-world testbeds and large-scale simulations—the study achieves end-to-end global optimization of computation and communication. Experimental results demonstrate that the proposed solution improves training throughput by up to 64.62% over current production configurations and by 37.59% compared to state-of-the-art baselines.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

Reinforcement learning (RL) training of large language models (LLMs) on heterogeneous GPU clusters suffers from low resource utilization due to significant disparities in computational intensity, memory demand, and communication patterns across the three sequential phases—rollout generation, reward computation, and policy update. Method: This paper proposes AReaL-Hex, a system featuring a fully asynchronous RL architecture that decouples these three phases, coupled with a two-stage scheduler integrating mixed-integer linear programming (MILP)-based constraint search and graph partitioning to dynamically assign HBM/I/O-intensive and compute-intensive tasks to optimal heterogeneous devices while preserving data freshness. Contribution/Results: Experiments on 1.5B–14B LLMs demonstrate that AReaL-Hex achieves a 1.50× higher training throughput under identical budget constraints and reduces training cost by 1.46× at equivalent throughput, significantly improving efficiency on heterogeneous GPU clusters.

To address challenges in distributed ML training across dynamic, heterogeneous, and geographically dispersed clusters—including inefficient resource allocation, poor throughput-cost trade-offs, and insufficient heterogeneity support in existing frameworks—this paper proposes a full-stack automated training system. The system integrates lightweight runtime/memory simulation, adaptive search-space pruning, and a heterogeneity-aware training framework. Key contributions include: (1) a performance-modeling–guided efficient configuration search algorithm; (2) a heterogeneity-compatible distributed training runtime; (3) a cross-Availability-Zone resource scheduling mechanism; and (4) dynamic topology-aware communication optimization. Evaluated on real-world heterogeneous clusters, the system achieves 2.1× higher training throughput and reduces per-task cost by 37% compared to state-of-the-art baselines. It converges to optimal configurations within minutes and requires zero manual hyperparameter tuning.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

Training large language models relies on GPU clusters comprising thousands of devices, resulting in prohibitively high development and debugging costs and making it difficult to reproduce production-scale training behavior. This work proposes PrismLLM, a framework that constructs a high-fidelity model of computation, communication, and dependencies by slicing the execution graph and integrates a hybrid simulation mechanism combining real execution with virtual replay. PrismLLM accurately emulates large-scale training dynamics on drastically reduced hardware—using fewer than 1% of the original GPUs—and successfully reproduces training behavior at an 8,192-GPU scale with only 0.58% error in iteration time and less than 0.01% error in peak GPU memory usage. This approach substantially lowers the barrier to entry for large-model development and significantly enhances debugging efficiency.

Existing large-scale model training systems struggle to flexibly compose diverse parallelization strategies, often relying on manual expert tuning and lacking generality. This work proposes a programmable distributed training system that enables users to declaratively specify composite parallelism strategies—such as data, pipeline, and expert parallelism—through model annotations and scheduling directives. These specifications are compiled via a unified intermediate representation (IR) into device-level execution plans, fully decoupling strategy definition from runtime execution over a global compute-communication DAG. The system is the first to support automatic compilation of user-defined composite strategies, matching the performance of established approaches like ZeRO while significantly improving both performance and memory efficiency in complex scenarios such as DeepSeek-V3’s DualPipe.

The surging computational demands of large language model (LLM) training exceed the capacity of homogeneous clusters from a single vendor, necessitating heterogeneous collaborative training across AMD and NVIDIA GPUs. This work proposes a compatibility-oriented hybrid communication architecture that integrates CPU-mediated forwarding and direct device-to-device connectivity, and introduces an innovative CPU-offloaded peer-to-peer transfer mechanism that enables cross-vendor GPU data exchange without host memory staging for the first time. By leveraging multi-NIC parallel transmission and differentiated communication backend scheduling, the system achieves 98% of the throughput of an NVIDIA-only homogeneous cluster on LLaMA-8B and Qwen2-7B models while preserving training stability and correctness.

This work addresses the absence of principled guidelines for optimal computational resource allocation in reinforcement learning (RL) post-training of large language models (LLMs). Focusing on the joint optimization of the number of parallel trajectories, problems per batch, and update steps under a fixed compute budget, the study uncovers a saturation effect wherein the optimal number of parallel trajectories plateaus as the budget increases, with distinct underlying mechanisms observed between easy and hard tasks. Within an online policy gradient RL framework, the authors conduct systematic ablations and validate findings across diverse base models and data distributions. They propose a practical, efficient resource allocation strategy that consistently enhances both sample and computational efficiency, offering actionable guidance for RL-based post-training of LLMs.

This work addresses the limitations of current robotic reinforcement learning systems, which rely heavily on GPU-based centralized simulation constrained by the CUDA ecosystem. The authors propose UniLab, a heterogeneous architecture that achieves the first efficient decoupling of CPU-parallelized simulation and GPU-accelerated policy learning. By introducing a unified runtime to manage data movement, buffering, and synchronization, UniLab establishes an end-to-end training loop across diverse hardware platforms. The framework supports non-CUDA environments—including macOS, ROCm, and Intel XPU—and integrates CPU-batched physics backends (MuJoCoUni and MotrixSim) alongside mainstream RL algorithms such as PPO, SAC, and TD3. Experimental results demonstrate a 3–10× improvement in training efficiency over existing approaches under identical hardware conditions, substantially overcoming prevailing platform and performance bottlenecks.