🤖 AI Summary
In large-scale distributed ML training, collective communication primitives (e.g., AllReduce, AllGather, Broadcast) generate high-bandwidth, bursty traffic, causing network congestion and packet loss. Method: This work systematically characterizes the communication behavior of mainstream LLMs—including DeepSeek-V3, GPT, and Llama—under diverse parallelism strategies, scales, and network topologies. Leveraging fine-grained empirical analysis of NVIDIA NCCL logs, we quantitatively identify how operation type, message size, and request distribution affect network anomalies. We propose a resource-coordinated optimization framework tailored to LLM communication patterns, jointly optimizing collective primitive scheduling and network topology adaptation. Contribution/Results: Experiments demonstrate that our approach significantly mitigates congestion, improving both communication efficiency and stability in distributed training and inference—without modifying model architecture or training algorithms.
📝 Abstract
Machine Learning jobs, carried out on large number of distributed high performance systems, involve periodic communication using operations like AllReduce, AllGather, and Broadcast. These operations may create high bandwidth and bursty traffic patterns, leading to network congestion and packet loss, thus impacting the performance of these jobs. Hence it is imperative to analyze these patterns, which can be helpful in provisioning network resources depending on the type of machine learning workloads. In this poster we carry out extensive analysis of the collective communication behavior seen in a wide variety of models (ex. DeepSeek, GPT, Llama, etc.) To achieve this we instrument Nvidia Collective Communication Library logging functionality for richer context about the collectives and workloads. We adjust configuration parameters that influence collective communication behavior, such as parallelism, number of nodes, and model type. This overview presents and discusses some of the results on the collective communication behavior for the open source DeepSeek V3 inferencing model, which includes operation type and count, transfer sizes per operation, and request size distribution. Our analysis shows that it makes sense to rethink current collective communication frameworks and network topologies so as to accommodate the effect of network anomalies on the mentioned workloads.