checkpoint management

In large-scale DNN distributed training, checkpointing is tightly coupled with model parallelism strategies and hardware topology, severely limiting fault tolerance and elastic scalability. To address this, we propose the “distributed storage, unified loading” paradigm: during saving, model parameters are stored in a distributed representation aligned with the current parallel configuration; during restoration, they are uniformly reconstructed into a logically consistent parameter view. We design a universal checkpoint format—incorporating merged parameter representations and mapping metadata—a Universal Checkpoint Language (UCL), and an on-demand state reconstruction mechanism, achieving, for the first time, full decoupling of checkpointing from parallel configurations. Evaluated on LLaMA, Bloom, and other mainstream large models under diverse parallelism paradigms—including tensor parallelism (TP), pipeline parallelism (PP), data parallelism (DP), and context parallelism (CP)—our approach reduces post-failure recovery time by 12–28% on average, significantly enhancing cross-configuration portability and system robustness.

Training large foundation models (LFMs) faces significant challenges in checkpoint management, including poor cross-framework compatibility, tight coupling with parallelization strategies, heterogeneous storage backends, and severe I/O bottlenecks. To address these, this work proposes an industrial-grade unified archival system. Its core contributions are: (1) a novel parallelism-agnostic checkpoint serialization format; (2) a full-stack I/O optimization framework integrating a dynamic resharding engine, multi-framework abstraction interfaces (PyTorch/Megatron/DeepSpeed), asynchronous high-throughput storage adapters, and a distributed I/O monitoring toolchain; and (3) runtime support for cross-parallelism resharding, multi-backend adaptivity, and rapid failure recovery. Experiments demonstrate an average 54.20× reduction in checkpoint blocking time, with peak checkpoint save and load speedups of 9.96× and 8.80×, respectively. The system has been stably deployed in production environments scaling to over one thousand GPUs.

To address the high checkpointing overhead and low fault tolerance efficiency in distributed training of ultra-large-scale sparse Mixture-of-Experts (MoE) models, this paper proposes MoC-System, a hybrid checkpointing system. Methodologically, it introduces (1) Partial Expert Checkpointing (PEC), a novel mechanism that selects expert subsets to achieve algorithm-system co-optimization; and (2) a two-level asynchronous checkpointing manager that decouples in-memory snapshotting from persistent storage, integrating fully sharded storage with ZeRO-2 and expert parallelism. Evaluated within the Megatron-DeepSpeed framework, MoC-System reduces per-checkpoint overhead by up to 98.9% while improving average downstream task accuracy by 1.08%—without any accuracy loss. This work constitutes the first systematic solution to efficient fault tolerance for highly scalable MoE model training.

Intermediate checkpoints in diffusion models (DMs) and consistency models (CMs) are often underutilized, despite evidence that optimal weights frequently reside in non-convex “basins” where SGD fails to converge. Method: We propose LCSC—a learning-based checkpoint selection and combination framework—that employs evolutionary search to automatically learn linear weighting coefficients over trajectory checkpoints, integrates multi-stage weights, and synergistically combines consistency distillation with diffusion sampling optimization. Contribution/Results: LCSC establishes a generalizable checkpoint-weighted averaging paradigm that improves both generation quality and inference efficiency without increasing computational cost at deployment. On CIFAR-10 and ImageNet-64, LCSC achieves up to 23× training speedup; reduces DM sampling NFE from 15 to 9; and enables CM single-step inference to outperform the two-step baseline—demonstrating for the first time that trajectory-weighted averaging can transcend SGD’s convergence limitations, thereby introducing a novel training paradigm for generative models.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

This work addresses the challenge of resource dynamics in shared clusters during large language model training, where conventional checkpoint-based state migration incurs substantial I/O overhead and GPU stalls. To overcome this, the authors propose ETC, a novel framework that eliminates checkpointing entirely by exploiting state locality and leveraging peer-to-peer GPU communication with communication aggregation to enable storage-free, low-fragmentation elastic state migration. Seamlessly integrated into the Megatron-LM hybrid parallel training system, ETC reduces migration overhead by 2.33–6.37× compared to checkpoint-based approaches across diverse parallelism configurations, significantly enhancing the practicality and efficiency of elastic training for large models.

This work addresses the challenge of frequent training interruptions in large language models caused by diverse system failures, which existing single-level checkpointing schemes struggle to mitigate without incurring high overhead or slow recovery. The authors propose a cluster-aware, three-tier hierarchical checkpointing architecture that aligns checkpoint placement with fault heterogeneity: lightweight differential checkpoints are stored in local or peer-node memory for rapid access, while heavyweight base checkpoints are asynchronously migrated to remote persistent storage. This design achieves, for the first time, cross-layer global consistency guarantees matched to fault types, enabling fast, cluster-informed recovery without blocking training progress. Experiments on a 40-billion-parameter model demonstrate total checkpointing times under 10 seconds, significantly reducing training overhead and supporting high-frequency checkpointing.

This work addresses the “scientific amnesia” problem in continual DPO training, where models retain prior behaviors yet fail to accumulate reusable methodological knowledge. We formalize this phenomenon as a measurable, industrial-scale challenge and introduce a comprehensive diagnostic framework comprising a procedural pipeline, a 30-round HumanEval subdomain benchmark, and five categories of strategy proposers. Leveraging chain-wise training of Qwen2.5-7B-Instruct with FSDP-sharded DPO checkpoints, we evaluate strategies including rule-based scheduling, retrieval-augmented memory, Bayesian optimization, and a meta-scientific reasoning model (MSCL). Experiments reveal that only rule-based scheduling improves performance in homogeneous single-seed chains, while MSCL uniquely enhances outcomes in heterogeneous chains. Across multi-seed settings, retrieval-augmented memory yields the highest average gains, though differences among methods do not reach statistical significance.

This work addresses the inefficiency in notebook-based distributed workflows, where minor modifications often trigger full re-execution, severely hindering iterative development and reproducibility. To overcome this limitation, the authors propose NBRewind, a system that, for the first time, enables fine-grained incremental execution and cross-platform portability while preserving reproducibility. NBRewind integrates a dual-kernel architecture—comprising auditing and replay components—with cell-level incremental checkpoints and inter-cell dataflow analysis. It further leverages standardized notebook packaging to facilitate efficient partial re-execution. Evaluation in real-world high-performance computing (HPC) scenarios demonstrates that NBRewind incurs minimal overhead for incremental checkpointing and substantially improves both execution efficiency and cross-site reproducibility.

Current quantum high-performance computing systems lack effective fault-tolerance and recovery mechanisms, and conventional checkpointing approaches based on quantum state preservation are fundamentally hindered by the no-cloning theorem. This work proposes a novel algorithm-level fault-tolerance framework that reframes checkpointing and recovery as problems of control flow and algorithmic state management, thereby avoiding direct storage of quantum states. Instead, it leverages mid-circuit measurements, classical feedforward, and conditional operations in dynamic quantum circuits to efficiently capture and restore program execution states. The approach enables reliable interruption and resumption for representative iterative or staged quantum algorithms—including variational eigensolvers, the Quantum Approximate Optimization Algorithm (QAOA), and time-stepping simulations—significantly enhancing the reliability and restartability of quantum computational tasks.