MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning

2025-11-10

AI Summary

Current multimodal large language models (MLLMs) exhibit limited generalization on complex mathematical reasoning tasks, primarily constrained by static knowledge distillation from fixed teacher models and static datasets. Method: We propose the Self-Evolving Iterative Reflection (SEIR) framework, which integrates multi-step reasoning, dynamic self-reflection, and reward-guided fine-tuning. A dedicated Outcome Reward Model (ORM) generates high-quality reflection signals to drive iterative refinement within a closed-loop reasoning–reflection–optimization cycle. Contribution/Results: SEIR transcends conventional data distillation paradigms by enabling deep generalization to unseen, complex problems without reliance on static supervision. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art open-source MLLMs—including QVQ—on rigorous benchmarks such as MathVL-test, validating both its effectiveness and conceptual novelty.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in vision-language answering tasks. Despite their strengths, these models often encounter challenges in achieving complex reasoning tasks such as mathematical problem-solving. Previous works have focused on fine-tuning on specialized mathematical datasets. However, these datasets are typically distilled directly from teacher models, which capture only static reasoning patterns and leaving substantial gaps compared to student models. This reliance on fixed teacher-derived datasets not only restricts the model's ability to adapt to novel or more intricate questions that extend beyond the confines of the training data, but also lacks the iterative depth needed for robust generalization. To overcome these limitations, we propose extbf{method}, a extbf{Math}ematical extbf{S}elf- extbf{E}volving framework for MLLMs. In contrast to traditional one-shot fine-tuning paradigms, method iteratively refines the model through cycles of inference, reflection, and reward-based feedback. Specifically, we leverage iterative fine-tuning by incorporating correct reasoning paths derived from previous-stage inference and integrating reflections from a specialized Outcome Reward Model (ORM). To verify the effectiveness of method, we evaluate it on a suite of challenging benchmarks, demonstrating significant performance gains over backbone models. Notably, our experimental results on MathVL-test surpass the leading open-source multimodal mathematical reasoning model QVQ. Our code and models are available at exttt{https://zheny2751allowbreak-dotcom.github.io/allowbreak MathSE.github.io/}.

Problem

• Multimodal models struggle with complex mathematical reasoning tasks
• Fixed teacher-derived datasets limit adaptation to novel questions
• Traditional fine-tuning lacks iterative depth for robust generalization

Innovation

• Iterative refinement through inference-reflection cycles
• Incorporating correct reasoning paths from previous stages
• Integrating feedback from specialized Outcome Reward Model