Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning

2025-05-18

AI Summary

To address the weak reasoning capabilities of multimodal large language models (MLLMs) and their difficulty adapting to diverse multimodal inputs and formats, this paper proposes a dynamic progressive reinforcement learning (RL) framework. Methodologically, we construct the NeuraLadder dataset—a hierarchically difficulty-graded benchmark—and introduce multimodal format constraints to strengthen vision-language alignment. We further design an uncertainty-aware dynamic weighting reward mechanism, incorporating a length-sensitive bonus and difficulty-adaptive weight scheduling. Our key contribution is the first formal modeling of human learning principles as a controllable, progressive RL paradigm for MLLMs. Extensive evaluation on Qwen2.5-VL variants demonstrates substantial gains over larger baseline models on ReasoningBench and MMMU, notably improving chain-of-thought clarity, conciseness, and accuracy. Ablation studies confirm the efficacy of each component.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Reinforcement Learning (RL) has shown promise in improving the reasoning abilities of Large Language Models (LLMs). However, the specific challenges of adapting RL to multimodal data and formats remain relatively unexplored. In this work, we present Observe-R1, a novel framework aimed at enhancing the reasoning capabilities of multimodal large language models (MLLMs). We draw inspirations from human learning progression--from simple to complex and easy to difficult, and propose a gradual learning paradigm for MLLMs. To this end, we construct the NeuraLadder dataset, which is organized and sampled according to the difficulty and complexity of data samples for RL training. To tackle multimodal tasks, we introduce a multimodal format constraint that encourages careful observation of images, resulting in enhanced visual abilities and clearer and more structured responses. Additionally, we implement a bonus reward system that favors concise, correct answers within a length constraint, alongside a dynamic weighting mechanism that prioritizes uncertain and medium-difficulty problems, ensuring that more informative samples have a greater impact on training. Our experiments with the Qwen2.5-VL-3B and Qwen2.5-VL-7B models on 20k samples from the NeuraLadder dataset show that Observe-R1 outperforms a series of larger reasoning models on both reasoning and general benchmarks, achieving superior clarity and conciseness in reasoning chains. Ablation studies validate the effectiveness of our strategies, highlighting the robustness and generalization of our approach. The dataset and code will be released at https://github.com/zrguo/Observe-R1.

Problem

• Enhancing reasoning in MLLMs with dynamic progressive RL
• Adapting RL for multimodal data complexity and formats
• Improving visual observation and structured response generation

Innovation

• Dynamic Progressive Reinforcement Learning for MLLMs
• Multimodal format constraint enhances visual abilities
• Bonus reward system prioritizes concise correct answers