Cross-Layer Energy Analysis of Multimodal Training on Grace Hopper Superchips

2026-05-03

AI Summary

This work addresses the energy inefficiency in training large multimodal models, which is primarily constrained by memory and communication bottlenecks stemming from data movement rather than computational underutilization. Through a cross-stack analysis on the NVIDIA Grace Hopper superchip, the study reveals that runtime-optimal configurations do not necessarily yield peak energy efficiency. To bridge this gap, the authors propose an energy-aware training strategy that leverages high-bandwidth CPU–GPU interconnects, unified memory, CPU offloading, activation checkpointing, sequence parallelism, and hardware-aware scheduling. This approach effectively balances energy efficiency, execution time, and throughput, offering practical guidelines for efficient multimodal model training on heterogeneous systems.

Abstract

Multimodal deep learning models enable joint learning across heterogeneous data sources, including text, images, and video, but their rapid scaling introduces significant memory and communication bottlenecks. As model sizes and sequence lengths increase, training performance becomes increasingly impacted by data movement rather than computation. Frameworks such as DeepSpeed mitigate these challenges through CPU offloading, activation checkpointing, and communication optimizations. However, these techniques introduce additional system activity, which may affect energy efficiency. Meanwhile, tightly integrated heterogeneous architectures, such as the NVIDIA Grace Hopper (GH200) superchip, provide high-bandwidth CPU-GPU interconnects and unified memory, thereby reducing data transfer overhead. In this work, we present a cross-layer analysis of energy and performance trade-offs in multimodal training on GH200 systems, explicitly characterizing the interactions between application, runtime, and hardware layers. Leveraging high-bandwidth CPU-GPU interconnects, our results show that energy efficiency is primarily governed by data movement and overlap rather than raw compute utilization, and that configurations optimized for runtime are not necessarily optimal for energy. Based on these findings, we distill a set of actionable guidelines for practitioners that demonstrate how to balance offloading strategies, sequence parallelism, and hardware-aware scheduling to achieve energy-efficient training. Our results demonstrate that leveraging high-bandwidth CPU-GPU interconnects enables offloading strategies and sequence parallelism, achieving a strong balance among energy efficiency, runtime performance, and computational throughput, providing practical guidelines for efficient multimodal training on modern heterogeneous systems.

Problem

• multimodal training
• energy efficiency
• data movement
• heterogeneous architectures
• Grace Hopper

Innovation

• cross-layer analysis
• energy efficiency
• multimodal training
• Grace Hopper
• data movement