Machine Learning Fleet Efficiency: Analyzing and Optimizing Large-Scale Google TPU Systems with ML Productivity Goodput

2025-02-10

AI Summary

Traditional hardware utilization metrics fail to reflect true efficiency in large-scale ML fleets (e.g., Google TPU clusters) due to deep, multi-layered software–hardware coupling across models, data pipelines, frameworks, compilers, and schedulers. Method: This paper introduces an end-to-end cluster productivity analysis framework grounded in production-grade TPU telemetry, workload characterization, hierarchical performance attribution, and actionable optimization recommendations. Contribution/Results: We propose ML Productivity Goodput (MPG), the first holistic, stack-wide efficiency metric explicitly designed to quantify productive throughput—spanning models, datasets, frameworks, compilers, and schedulers—thereby transcending conventional hardware-centric utilization measures. Evaluated on real internal workloads, MPG enables precise, cross-layer bottleneck identification and drives measurable improvements in fleet effective throughput and resource return on investment.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Recent years have seen the emergence of machine learning (ML) workloads deployed in warehouse-scale computing (WSC) settings, also known as ML fleets. As the computational demands placed on ML fleets have increased due to the rise of large models and growing demand for ML applications, it has become increasingly critical to measure and improve the efficiency of such systems. However, there is not yet an established methodology to characterize ML fleet performance and identify potential performance optimizations accordingly. This paper presents a large-scale analysis of an ML fleet based on Google's TPUs, introducing a framework to capture fleet-wide efficiency, systematically evaluate performance characteristics, and identify optimization strategies for the fleet. We begin by defining an ML fleet, outlining its components, and analyzing an example Google ML fleet in production comprising thousands of accelerators running diverse workloads. Our study reveals several critical insights: first, ML fleets extend beyond the hardware layer, with model, data, framework, compiler, and scheduling layers significantly impacting performance; second, the heterogeneous nature of ML fleets poses challenges in characterizing individual workload performance; and third, traditional utilization-based metrics prove insufficient for ML fleet characterization. To address these challenges, we present the"ML Productivity Goodput"(MPG) metric to measure ML fleet efficiency. We show how to leverage this metric to characterize the fleet across the ML system stack. We also present methods to identify and optimize performance bottlenecks using MPG, providing strategies for managing warehouse-scale ML systems in general. Lastly, we demonstrate quantitative evaluations from applying these methods to a real ML fleet for internal-facing Google TPU workloads, where we observed tangible improvements.

Problem

• Measure and improve ML fleet efficiency.
• Identify ML fleet performance bottlenecks.
• Optimize large-scale ML systems effectively.

Innovation

• Introduces ML Productivity Goodput metric
• Analyzes efficiency across ML system stack
• Optimizes performance bottlenecks using MPG