Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection

2026-06-02

AI Summary

Existing token entropy–based reinforcement learning methods underperform in visual reasoning because they overlook visually salient yet low-entropy tokens, failing to balance perceptual grounding with semantic reasoning. This work proposes the VEPO framework, which explicitly couples visual saliency with token entropy to establish a credit assignment mechanism tailored for visual reasoning. By multiplicatively fusing these signals, VEPO directs gradient updates toward tokens that are both visually anchored and semantically informative. This approach overcomes the limitations of purely entropy-driven exploration, achieving performance gains of 2.28 and 3.15 points over entropy-based baselines on 7B and 3B models, respectively, thereby substantially enhancing visual reasoning capabilities.

Abstract

While token-level entropy is commonly recognized as effective for credit assignment in text-only reinforcement learning with verifiable rewards (RLVR), it remains unclear whether this mechanism still holds in visual reasoning. Our controlled study shows that this mechanism collapses in visual reasoning due to the omission of vision-sensitive tokens with naturally low entropy. Although existing multimodal RL methods increasingly acknowledge the importance of visual perception, they struggle to satisfy the inherent demand for interleaving precise perceptual grounding with semantic reasoning, either lacking systematic visual measurements or overlooking that token entropy primarily drives semantic exploration. To address this, we introduce VEPO (Vision-Entropy token-selection for Policy Optimization), an effective RL framework explicitly integrating visual sensitivity with token entropy via a principled multiplicative coupling, where VEPO redirects gradient credit toward tokens which are simultaneously visually grounded and highly informative. Extensive experiments demonstrate VEPO's leading performance, significantly outperforming the entropy-only baseline by 2.28 points at 7B-scale and 3.15 points at 3B-scale. Ablations further substantiate the soundness of our method.

Problem

• visual reasoning
• reinforcement learning
• token entropy
• credit assignment
• multimodal perception

Innovation

• Vision-Entropy coupling
• token selection
• visual reasoning
• reinforcement learning
• credit assignment