PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment

2026-02-28

AI Summary

Existing multimodal reinforcement learning approaches focus solely on optimizing answer correctness while overlooking erroneous interpretations of visual evidence during reasoning—referred to as process hallucination. To address this, this work proposes PaLMR, a novel framework that achieves the first joint alignment between visual perception and logical reasoning chains. PaLMR constructs structured pseudo-ground-truth labels and verifiable visual facts, and introduces a hierarchical reward mechanism based on process-level scoring to align both reasoning outcomes and intermediate steps at the data and optimization levels simultaneously. Experimental results demonstrate that PaLMR significantly suppresses process hallucination when applied to Qwen2.5-VL-7B, achieving state-of-the-art performance on HallusionBench while maintaining leading accuracy on benchmarks such as MMMU, MathVista, and MathVerse.

Abstract

Reinforcement learning has recently improved the reasoning ability of Large Language Models and Multimodal LLMs, yet prevailing reward designs emphasise final-answer correctness and consequently tolerate process hallucinations--cases where models reach the right answer while misperceiving visual evidence. We address this process-level misalignment with PaLMR, a framework that aligns not only outcomes but also the reasoning process itself. PaLMR comprises two complementary components: a perception-aligned data layer that constructs process-aware reasoning data with structured pseudo-ground-truths and verifiable visual facts, and a process-aligned optimisation layer that constructs a hierarchical reward fusion scheme with a process-aware scoring function to encourage visually faithful chains-of-thought and improve training stability. Experiments on Qwen2.5-VL-7B show that our approach substantially reduces reasoning hallucinations and improves visual reasoning fidelity, achieving state-of-the-art results on HallusionBench while maintaining strong performance on MMMU, MathVista, and MathVerse. These findings indicate that PaLMR offers a principled and practical route to process-aligned multimodal reasoning, advancing the reliability and interpretability of MLLMs.

Problem

• visual reasoning
• process hallucination
• multimodal alignment
• reasoning fidelity
• MLLMs

Innovation

• process alignment
• multimodal reasoning
• reasoning hallucination
• reinforcement learning
• visual fidelity