Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency

2025-12-08

AI Summary

Existing studies lack a systematic characterization of the coupled computational, communication, memory, and power behaviors in multi-GPU large language model (LLM) training. This paper introduces Chopper—the first full-stack, multi-granularity (kernel-/layer-/stage-/iteration-level) performance analysis framework tailored for the AMD Instinct™ MI300X platform. Chopper integrates hardware performance counters with kernel-level trace alignment to enable end-to-end behavioral modeling. Our analysis reveals, for the first time, that DVFS frequency adjustment overhead—not MFMA underutilization or insufficient communication-computation overlap—is the dominant bottleneck in Llama 3 8B training. Furthermore, we identify memory access determinism as a critical factor governing GPU frequency stability. These findings provide empirically grounded, actionable insights for optimizing distributed training frameworks and guiding AI accelerator architecture design.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsComputer Vision: Large Vision Models

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Training large language models (LLMs) efficiently requires a deep understanding of how modern GPU systems behave under real-world distributed training workloads. While prior work has focused primarily on kernel-level performance or single-GPU microbenchmarks, the complex interaction between communication, computation, memory behavior, and power management in multi-GPU LLM training remains poorly characterized. In this work, we introduce Chopper, a profiling and analysis framework that collects, aligns, and visualizes GPU kernel traces and hardware performance counters across multiple granularities (i.e., from individual kernels to operations, layers, phases, iterations, and GPUs). Using Chopper, we perform a comprehensive end-to-end characterization of Llama 3 8B training under fully sharded data parallelism (FSDP) on an eight-GPU AMD InstinctTM MI300X node. Our analysis reveals several previously underexplored bottlenecks and behaviors, such as memory determinism enabling higher, more stable GPU and memory frequencies. We identify several sources of inefficiencies, with frequency overhead (DVFS effects) being the single largest contributor to the gap between theoretical and observed performance, exceeding the impact of MFMA utilization loss, communication/computation overlap, and kernel launch overheads. Overall, Chopper provides the first holistic, multi-granularity characterization of LLM training on AMD InstinctTM MI300X GPUs, yielding actionable insights for optimizing training frameworks, improving power-management strategies, and guiding future GPU architecture and system design.

Problem

• Characterizes multi-GPU LLM training inefficiencies across communication, computation, and power
• Identifies frequency overhead as the primary performance gap contributor in distributed training
• Provides holistic profiling to optimize training frameworks and GPU system design

Innovation

• Multi-granularity profiling framework for GPU performance analysis
• Identifies frequency overhead as major bottleneck in LLM training
• Provides holistic characterization of multi-GPU LLM training inefficiencies