Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training

2026-08-04

AI Summary

This work addresses catastrophic forgetting in multimodal large language models during continual reinforcement fine-tuning, which stems from task distribution shifts and the limitations of existing methods that rely on implicit reward variance regularization to control optimization risk. To mitigate this, the authors propose Risk-Aware Policy Optimization (RAPO), a dual-channel framework that explicitly manages risk. The policy channel adaptively modulates update magnitudes by integrating rollback reliability with Fisher information–inspired local sensitivity, while the data channel dynamically reorganizes batches through risk-aware stratification to prioritize informative and stable samples. RAPO requires no cross-task memory, is plug-and-play compatible with any reinforcement fine-tuning algorithm, and achieves a 79.8% reduction in forgetting rate over the RLOO baseline on the MLLM-CL benchmark while preserving performance on new tasks.

Abstract

Reinforcement fine-tuning (RFT) is widely believed to inherently resist catastrophic forgetting in continual post-training of multimodal large language models. Under pronounced task distributional shifts, however, forgetting across representative RFT algorithms escalates sharply. This stems from the implicit reward-variance regularization inherent to RFT, which proves incapable of suppressing uncontrolled optimization risk. We propose Risk-Aware Policy Optimization (RAPO), the first dual-channel framework for explicit risk governance in continual RFT. On the policy channel, Risk-Aware Policy Scaling adaptively calibrates per-sample update magnitude via rollout reliability and Fisher-inspired local predictive sensitivity; on the data channel, Risk-Aware Dynamic Bucket Sampling reorganizes training batches through dynamic risk stratification, steering optimization toward informative yet stable samples. As a plug-and-play strategy requiring no cross-task memory, RAPO generalizes to any RFT algorithm without modification. On the public MLLM-CL benchmark, RAPO reduces final forgetting by 79.8% relative to its RLOO backbone while retaining new-task competitiveness.

Problem

• catastrophic forgetting
• reinforcement fine-tuning
• multimodal large language models
• distributional shift
• optimization risk

Innovation

• Risk-Aware Policy Optimization
• Dual-Channel Framework
• Continual Reinforcement Fine-Tuning
• Catastrophic Forgetting Mitigation
• Dynamic Risk Stratification