Role-Based Fault Tolerance System for LLM RL Post-Training

2025-12-27

AI Summary

During RL post-training of large language models, concurrent training and inference execution introduces dual-failure risks; existing fault-tolerance frameworks lack isolation and coordinated recovery support for asynchronous roles (trainer/rollout). Method: We propose a role-isolation-based fine-grained fault-tolerance mechanism: (i) a novel role-level fault isolation and dynamic reconnection architecture enabling non-interruptive trainer recovery, hot replacement of rollout machines, and UCX-based point-to-point weight synchronization; (ii) role-aware monitoring, warm standby, state-persistent trajectory generation, and an asynchronous reconnection protocol. Results: Evaluated on the Qwen3-8B-Math task across a 256-GPU cluster, our approach achieves an effective training time ratio of 80%—a 20-percentage-point improvement over ByteRobust—and delivers end-to-end speedups of 8.4%–17.4%, significantly surpassing conventional full-restart paradigms.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Multiagent Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

RL post-training for LLMs has been widely scaled to enhance reasoning and tool-using capabilities. However, RL post-training interleaves training and inference workloads, exposing the system to faults from both sides. Existing fault tolerance frameworks for LLMs target either training or inference, leaving the optimization potential in the asynchronous execution unexplored for RL. Our key insight is role-based fault isolation so the failure in one machine does not affect the others. We treat trainer, rollout, and other management roles in RL training as distinct distributed sub-tasks. Instead of restarting the entire RL task in ByteRobust, we recover only the failed role and reconnect it to living ones, thereby eliminating the full-restart overhead including rollout replay and initialization delay. We present RobustRL, the first comprehensive robust system to handle GPU machine errors for RL post-training Effective Training Time Ratio improvement. (1) extit{Detect}. We implement role-aware monitoring to distinguish actual failures from role-specific behaviors to avoid the false positive and delayed detection. (2) extit{Restart}. For trainers, we implement a non-disruptive recovery where rollouts persist state and continue trajectory generation, while the trainer is rapidly restored via rollout warm standbys. For rollout, we perform isolated machine replacement without interrupting the RL task. (3) extit{Reconnect}. We replace static collective communication with dynamic, UCX-based (Unified Communication X) point-to-point communication, enabling immediate weight synchronization between recovered roles. In an RL training task on a 256-GPU cluster with Qwen3-8B-Math workload under 10% failure injection frequency, RobustRL can achieve an ETTR of over 80% compared with the 60% in ByteRobust and achieves 8.4%-17.4% faster in end-to-end training time.

Problem

• Handles GPU machine faults in RL post-training for LLMs
• Isolates failures by role to avoid full system restarts
• Enables dynamic recovery and reconnection of failed components

Innovation

• Role-based fault isolation for distinct RL sub-tasks
• Dynamic UCX-based communication for weight synchronization
• Non-disruptive recovery with warm standbys and isolated replacement