MoFa: A Unified Performance Modeling Framework for LLM Pretraining

2025-11-13

AI Summary

Existing performance modeling approaches for distributed pretraining of trillion-parameter large language models across thousands of devices fail to accurately predict training time, primarily because they neglect critical optimization characteristics and fault-tolerance overhead—especially checkpoint-based recovery costs. Method: This paper introduces the first unified performance modeling framework that jointly captures hybrid parallelism strategies, multi-dimensional system-level optimization features, and failure-recovery overhead estimated from historical reliability data. It integrates an enhanced cost model with an end-to-end tuning system. Contribution/Results: The framework achieves high-accuracy, long-horizon performance prediction (mean error <8%) and systematic bottleneck analysis for extended pretraining runs. Experimental evaluation demonstrates its capability to identify dominant factors limiting pretraining efficiency, thereby providing reliable, prior-guided insights for the design and deployment of large-model infrastructure.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Scalability of ML Systems
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The exponential growth in LLM scales, with parameters soaring from billions to trillions, has necessitated distributed pretraining across large clusters comprising thousands to tens of thousands of devices. While hybrid parallelization strategies enable such pretraining, the vast combinatorial strategy space introduces significant optimization challenges. Traditional manual tuning methods incur prohibitive trial-and-error costs, and existing performance modeling approaches exhibit critical limitations: they fail to comprehensively account for prevalent optimization features and ignore the substantial overhead imposed by essential fault tolerance mechanisms like checkpoint recovery in long-duration pretraining. To address these gaps, we propose MoFa, a novel pretraining performance modeling framework that unifies multi-dimensional optimization features and fault tolerance. MoFa incorporates an enhanced cost model to accurately capture the effects of key optimizations and integrates a fault tolerance model based on historical cluster reliability data. Besides, a MoFa-based tuning system is developed to explore optimal pretraining performance and potential bottlenecks in various scenarios. Extensive modeling evaluations demonstrate that MoFa can achieve high prediction accuracy across various scenarios. In addition, through comprehensive tuning experiments, our framework systematically reveals the key factors influencing pretraining performance under different configurations, which provides solid a priori guidance for LLM pretraining system design and deployment.

Problem

• Modeling distributed LLM pretraining performance across large clusters
• Addressing optimization challenges in hybrid parallelization strategy space
• Incorporating fault tolerance mechanisms into performance prediction models

Innovation

• Unifies multi-dimensional optimization features and fault tolerance
• Incorporates enhanced cost model for key optimizations
• Integrates fault tolerance model using historical reliability data