Optimizing Effective Training Time for Large-Scale Recommendation Systems

2026-10-01

AI Summary

This study addresses the problem of accelerator resource wastage caused by lifecycle overheads during large-scale recommendation system training. To mitigate this, it proposes an effective training time framework alongside an end-to-end overhead quantification and attribution methodology. By identifying full-stack redundancies across communication, compilation, and checkpointing, the approach achieves cross-component co-optimization through techniques including PyTorch compilation caching, asynchronous checkpointing, pipeline overlapping, and dynamic shape handling. Benchmark evaluations demonstrate that the proposed method yields an average performance improvement of 15.5%, attains a peak utilization load of 85%, and achieves an overall cluster efficiency exceeding 90%.

Abstract

Lifecycle overhead silently consumes accelerator capacity across large-scale recommendation training fleets. Our largest recommendation workloads process tens of billions train- ing examples per day on thousands of GPUs. Before this work, only 50-60% of their end-to-end wall time advanced training on new data. We present a fleet-scale study of this lifecycle overhead and a set of optimizations spanning the full training stack. We use Effective Training Time (ETT%) as an operational framework to instrument lost time, localize it to independently owned infrastructure components, and expose work repeated across job restarts. This analysis guides optimizations like communication elimination and pipeline overlap during trainer initialization; dynamic-shape handling, autotuning pruning, and reusable Py- Torch 2 compilation caches; asynchronous checkpointing; stan- dalone model publishing; and reductions in recovery cost. We evaluate the optimizations on representative models and measure their impacts in our training fleet. ETT% improves on every benchmark, by 15.5% on average, and reaches 85% on our largest workload. Fleet-wide ETT% rose from about 80% to above 90% after deployment.

Problem

• Large-Scale Recommendation Systems
• Lifecycle Overhead
• Effective Training Time
• GPU Utilization

Innovation

• Effective Training Time
• Lifecycle Overhead
• Recommendation Systems
• PyTorch Compilation Cache
• Asynchronous Checkpointing