MM-Eureka: Exploring Visual Aha Moment with Rule-based Large-scale Reinforcement Learning

2025-03-10

AI Summary

This work addresses key challenges in multimodal reasoning—namely, the difficulty of applying reinforcement learning (RL), low data efficiency, and poor interpretability—by proposing the first rule-based RL paradigm tailored for vision-language joint tasks. Methodologically, it integrates multimodal alignment modeling, reward shaping, and a self-supervised reflection mechanism, enabling RL fine-tuning of both pretrained and instruction-tuned models without supervised fine-tuning. Experiments demonstrate that the model spontaneously develops reflective behavior during multimodal reasoning: accuracy improves progressively, response length increases, and—for the first time in visual space—the “Aha Moment” phenomenon, previously observed only in text-based RL, is replicated. Compared to supervised fine-tuning, the approach achieves significantly higher data efficiency. The project fully open-sources code, models, and datasets to advance standardized research in multimodal RL.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We present MM-Eureka, a multimodal reasoning model that successfully extends large-scale rule-based reinforcement learning (RL) to multimodal reasoning. While rule-based RL has shown remarkable success in improving LLMs' reasoning abilities in text domains, its application to multimodal settings has remained challenging. Our work reproduces key characteristics of text-based RL systems like DeepSeek-R1 in the multimodal space, including steady increases in accuracy reward and response length, and the emergence of reflection behaviors. We demonstrate that both instruction-tuned and pre-trained models can develop strong multimodal reasoning capabilities through rule-based RL without supervised fine-tuning, showing superior data efficiency compared to alternative approaches. We open-source our complete pipeline to foster further research in this area. We release all our codes, models, data, etc. at https://github.com/ModalMinds/MM-EUREKA

Problem

• Extends rule-based RL to multimodal reasoning.
• Develops multimodal reasoning without supervised fine-tuning.
• Demonstrates superior data efficiency in multimodal RL.

Innovation

• Extends rule-based RL to multimodal reasoning
• Develops multimodal capabilities without supervised fine-tuning
• Open-sources complete pipeline for further research