hardware-aware tuning

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

Existing large-scale model training systems struggle to flexibly compose diverse parallelization strategies, often relying on manual expert tuning and lacking generality. This work proposes a programmable distributed training system that enables users to declaratively specify composite parallelism strategies—such as data, pipeline, and expert parallelism—through model annotations and scheduling directives. These specifications are compiled via a unified intermediate representation (IR) into device-level execution plans, fully decoupling strategy definition from runtime execution over a global compute-communication DAG. The system is the first to support automatic compilation of user-defined composite strategies, matching the performance of established approaches like ZeRO while significantly improving both performance and memory efficiency in complex scenarios such as DeepSeek-V3’s DualPipe.

To address slow training on power-constrained low-power edge devices, this paper proposes a cross-layer joint optimization framework that co-optimizes GPU frequency and batch size. The method integrates a batch-size efficiency prediction model with dynamic, real-time device power measurement modeling, enabling simultaneous minimization of both training time and energy consumption under strict power budgets—a first in the literature. Through system-level parameter coordination and hardware-in-the-loop runtime optimization, it achieves a 2.4× speedup in training latency and substantial energy reduction on real edge platforms, without compromising model accuracy. The core innovation lies in deeply embedding hardware power characteristics into the training configuration decision loop, thereby transcending conventional static hyperparameter tuning paradigms.

This work addresses the substantial accelerator memory consumption of model parameters, gradients, and optimizer states in standard mixed-precision training, which hinders the scalability of large models. The authors propose a memory-efficient training method that significantly reduces quantization error in 8-bit optimizer states through compact master weight partitioning and a novel compression-expansion function. By integrating 16-bit gradients, an improved weight splitting strategy, and a gradient checkpointing mechanism, the approach remains compatible with mainstream optimizers such as SGD, AdamW, and Lion. The method reduces AdamW’s per-parameter memory footprint from 16 bytes to 7 bytes (or 5 bytes when gradients are released) and halves model checkpoint size, achieving lossless training quality across multiple vision and language benchmark tasks.

Despite growing interest in Mixture-of-Experts (MoE) models, large-scale end-to-end pretraining of MoE architectures on pure AMD hardware—specifically the MI300X GPU and Pollara interconnect—remains unexplored, raising questions about the platform’s readiness for state-of-the-art LLM training. Method: We introduce MI300X-aware Transformer module sizing guidelines, conduct full-stack Pollara communication microbenchmarks, and integrate optimized All-reduce/Reduce-scatter primitives, fault-tolerant training, and checkpoint reshaping. Contribution/Results: We successfully complete end-to-end pretraining of the ZAYA1-base MoE model (760M activated parameters, 8.3B total parameters) on native AMD infrastructure. Evaluated on reasoning, mathematics, and code generation, ZAYA1-base substantially outperforms Llama-3-8B and OLMoE, matching the performance of Qwen3-4B and Gemma3-12B. This demonstrates, for the first time, that AMD’s hardware-software stack achieves full maturity for high-performance MoE model training.

This work addresses the computational and memory bottlenecks that hinder efficient scaling in large model training. To overcome the limitations of conventional point-wise optimizations, the authors propose a throughput-centric strategy that systematically integrates multiple techniques: optimized data loading (OVERLORD), CPU memory offloading (DeepSpeed ZeRO-Offload), distributed compilation (Triton-distributed), and hardware-level dynamic voltage and frequency scaling (DVFS). This holistic approach achieves a 4.5% improvement in end-to-end training throughput, substantially reduces training costs, and enables efficient training of models significantly larger than the memory capacity of a single GPU.

Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hindered by large memory footprints, frequent large-scale communication across heterogeneous networks, and severe workload imbalance. To characterize these challenges, we develop a mathematical model that quantifies memory, compute, and communication requirements for MoE configurations under various parallelization schemes, verified through micro-benchmarking, code instrumentation, and hardware profiling. Our analysis identifies performance bottlenecks: all-to-all latency at scale from expert parallelism, insufficient compute-communication overlap, low GPU utilization from imbalanced skinny GEMMs, and the absence of platform-aware hybrid parallelization strategies. To address these, we introduce Piper, a framework that leverages resource modeling to identify efficient training strategies for MoE models on target HPC platforms, applying pipeline parallelism with optimized schedules. Piper achieves 2-3.5X higher MFU than state-of-the-art frameworks such as X-MoE, and a novel all-to-all algorithm delivers 1.2-9X bandwidth over vendor implementation.

Existing hardware-software co-design tools struggle to accurately model memory consumption and backward-pass complexity in neural network training. This work proposes the first extension of the experimentally validated inference modeling framework, Stream, to the training domain, introducing a comprehensive framework for modeling and optimizing training on heterogeneous dataflow accelerators. The framework supports training workflow modeling, exploration of layer fusion configurations, and optimization of activation checkpointing strategies. Integrated with a genetic algorithm for hardware architecture search, it is validated on ResNet-18 and a small-scale GPT-2 model, effectively uncovering critical trade-offs between performance and memory in training-specific hardware design and identifying superior architectures and training strategies.

This work addresses the need for efficient performance evaluation of large language model inference across diverse hardware and scheduling configurations, a task hindered by the high cost and poor generalizability of end-to-end empirical measurements. The authors propose a fine-grained inference simulator that decomposes computation kernels, communication, and host overheads using a roofline model and employs a discrete-event scheduler to accurately model optimizations such as prefix caching and continuous batching, enabling token-level execution tracing and kernel-level latency breakdown. Innovatively, a two-tier prediction mechanism is introduced, which achieves high-fidelity latency prediction across GPU generations without requiring target-device measurements or with only a single model-agnostic microbenchmark. Experiments demonstrate median errors of just 15.4% for time-to-first-token (TTFT), 12.8% for time-per-output-token (TPOT), and 3.0% for throughput—approaching the accuracy of specialized profiling tools while substantially reducing measurement overhead.