collective communication optimization

In large-scale distributed ML training, collective communication primitives (e.g., AllReduce, AllGather, Broadcast) generate high-bandwidth, bursty traffic, causing network congestion and packet loss. Method: This work systematically characterizes the communication behavior of mainstream LLMs—including DeepSeek-V3, GPT, and Llama—under diverse parallelism strategies, scales, and network topologies. Leveraging fine-grained empirical analysis of NVIDIA NCCL logs, we quantitatively identify how operation type, message size, and request distribution affect network anomalies. We propose a resource-coordinated optimization framework tailored to LLM communication patterns, jointly optimizing collective primitive scheduling and network topology adaptation. Contribution/Results: Experiments demonstrate that our approach significantly mitigates congestion, improving both communication efficiency and stability in distributed training and inference—without modifying model architecture or training algorithms.

Existing direct-connect network topologies exhibit inadequate adaptability to diverse scales, node degrees, and latency–bandwidth trade-offs in high-performance computing (HPC) collective communication. Method: This paper proposes an iterative expansion framework grounded in small-scale optimal base topologies. It introduces, for the first time, a graph-synthesis-driven automatic topology generation mechanism and designs the first polynomial-time collective communication scheduling algorithm for canonical large-scale topologies—including Dragonfly and Fat-Tree. Contribution/Results: The work unifies topology synthesis and scheduling optimization within a single modeling framework, enabling cross-platform deployment and large-scale simulation validation. Experimental evaluation demonstrates that the proposed approach reduces average communication latency by 23.6% and improves bandwidth utilization by 31.4% compared to conventional topologies, while significantly enhancing scalability and practical applicability in real-world HPC systems.

To address throughput bottlenecks in collective communication (e.g., all-reduce) during large language model training over heterogeneous interconnects—such as hybrid switch fabrics and direct-attached accelerators—this paper proposes the first theoretically optimal broadcast/aggregation tree scheduling framework supporting arbitrary topologies with strongly polynomial-time solvability. Our method constructs throughput-optimal spanning trees via graph-theoretic modeling, establishes a topology-agnostic universal scheduling formulation, and introduces a lightweight runtime adaptation layer. Evaluated on AMD MI250 and NVIDIA DGX A100 platforms, the framework achieves significant throughput improvements over RCCL/NCCL, yielding measurable LLM training acceleration. Moreover, its scheduling generation is both faster and higher-quality than current state-of-the-art methods, uniquely combining provable optimality with practical deployability.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

Existing schedulers for distributed machine learning jobs in multi-tenant torus-topology clusters struggle to simultaneously optimize communication efficiency—constrained by task shape—and cluster utilization, often facing an inherent trade-off between the two. This paper proposes the first joint dynamic adaptation framework that co-optimizes task shape and optical circuit-switched topology. By identifying isomorphic task shapes and enabling real-time topology reconfiguration, our approach jointly satisfies job placement and communication requirements while minimizing network contention and maximizing resource utilization. Evaluations on a 4,096-node torus cluster simulator demonstrate that our method improves absolute cluster utilization by 57% and reduces job completion time by up to 11×, significantly surpassing the performance limits of conventional schedulers.

This work addresses the network congestion and high latency in large model training and inference caused by redundant unicast transmissions in many-to-many collective communications such as AllGather and AlltoAll. To this end, the authors propose MultiWrite, a novel many-to-many transport semantic tailored for AI workloads. MultiWrite introduces a lightweight adaptation of multicast principles to collective communication, avoiding the management overhead and compatibility issues of traditional multicast while significantly reducing communication costs through hardware-aware multicast scheduling and redundancy elimination. A prototype implementation on Ascend NPUs demonstrates up to 33% reduction in collective communication latency on commercial hardware, with long-term stress tests confirming its stability and effectiveness.

This work addresses the challenge of balancing overhead and performance in reconfigurable optical circuit-switched networks, where collective communication is hindered by high reconfiguration latency and the inability to reuse optical links across communication steps. Focusing on All-to-All and AllReduce primitives prevalent in AI/ML and HPC workloads, the paper introduces a Bruck-based reusable subring mechanism that enables sustained multi-step reuse of optical links for the first time, effectively amortizing reconfiguration costs over multiple communication phases. By integrating sparse reconfiguration with a subring topology maintenance strategy, the system achieves efficient scheduling under millisecond-scale reconfiguration delays. Experiments demonstrate that All-to-All completion time improves by 3–10× over static baselines, while AllReduce outperforms existing approaches by up to 1.5× and surpasses the bandwidth-optimal Ring algorithm by 1.5–6.6× on small-to-medium scale tasks.

Collective communication in distributed machine learning often becomes a performance bottleneck due to the neglect of physical network topology and process group structure. This work proposes a scalable and general framework for synthesizing collective communication algorithms that, for the first time, incorporates process-group awareness into algorithm generation, supporting arbitrary communication patterns. By integrating topology-aware modeling with optimized search strategies, the framework automatically generates high-performance communication algorithms tailored to the actual process groups and underlying network topology. Experimental results demonstrate that the framework can synthesize an All-to-All algorithm for a 512-NPU system within 11.68 minutes, achieving performance close to the theoretical optimum.

This work addresses the challenge of efficiently exploring the vast and physically constrained design space of cross-layer heterogeneous systems to support mixed AI and high-performance computing (HPC) workloads. To this end, the authors propose CHASE, a novel framework that decouples hardware architecture design from task mapping. CHASE leverages hierarchical type graphs for system modeling, a topology-aware mapper, and a telemetry-guided optimizer to enable application-driven architecture search under deployment constraints. Experimental results demonstrate that CHASE achieves geometric mean speedups of 6.20× and 2.12× on sparse computing and large language model workloads, respectively, while reducing mapping time by 60.5% on average and converging to near-global-optimal solutions within 64 iterations.

This work addresses the performance bottlenecks caused by communication complexity in large-scale heterogeneous chiplet systems by proposing a topology-agnostic dynamic computation migration framework. Instead of merely relocating data, the framework innovatively migrates entire computational contexts—including both code and associated data—to more favorable locations. It integrates a multi-bandwidth-domain chiplet architecture, a hierarchical routing mechanism, and a lightweight machine learning–assisted traffic prediction and scheduling strategy to enable communication-aware load placement and adaptive routing optimization. Experimental results demonstrate migration success rates of 75.2%–97.9%, average latency reductions of 16.4%–62.5%, and up to a 12.5× improvement in throughput. Under large language model (LLM) workloads, the system achieves average improvements of 4.9× in execution time, 5.9× in throughput, and 1.8× in energy efficiency.