Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

2024-11-27

arXiv.org

AI Summary

Vision-language models (VLMs) continue to suffer from hallucination and reasoning-path deviation in multimodal reasoning. To address this, we propose an actor–critic-style decoupled framework: a Reasoner module generates stepwise reasoning paths, while a Critic module delivers fine-grained, context-sensitive, natural-language feedback—marking the first adaptation of reinforcement learning’s critic mechanism to multimodal reasoning and replacing scalar rewards with a text-based, preference-optimized Critic. Our method integrates Direct Preference Optimization (DPO), rule-guided preference data construction, dynamic textual policy fine-tuning, and a dual-module collaborative inference architecture. Evaluated on eight benchmarks, our approach outperforms GPT-4V on five, achieving significant gains in reasoning accuracy, computational efficiency, and real-world reliability—particularly in safety-critical applications such as autonomous driving and embodied AI.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Vision-language models (VLMs) have shown remarkable advancements in multimodal reasoning tasks. However, they still often generate inaccurate or irrelevant responses due to issues like hallucinated image understandings or unrefined reasoning paths. To address these challenges, we introduce Critic-V, a novel framework inspired by the Actor-Critic paradigm to boost the reasoning capability of VLMs. This framework decouples the reasoning process and critic process by integrating two independent components: the Reasoner, which generates reasoning paths based on visual and textual inputs, and the Critic, which provides constructive critique to refine these paths. In this approach, the Reasoner generates reasoning responses according to text prompts, which can evolve iteratively as a policy based on feedback from the Critic. This interaction process was theoretically driven by a reinforcement learning framework where the Critic offers natural language critiques instead of scalar rewards, enabling more nuanced feedback to boost the Reasoner's capability on complex reasoning tasks. The Critic model is trained using Direct Preference Optimization (DPO), leveraging a preference dataset of critiques ranked by Rule-based Reward~(RBR) to enhance its critic capabilities. Evaluation results show that the Critic-V framework significantly outperforms existing methods, including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner and constructive feedback from the preference-optimized Critic enables a more reliable and context-sensitive multimodal reasoning process. Our approach provides a promising solution to enhance the reliability of VLMs, improving their performance in real-world reasoning-heavy multimodal applications such as autonomous driving and embodied intelligence.

Problem

• Enhance VLM accuracy in multimodal reasoning tasks
• Address hallucinated image understandings and unrefined reasoning paths
• Improve reliability of VLMs for real-world applications

Innovation

• Decouples reasoning and critic processes
• Uses reinforcement learning with natural language critiques
• Trains Critic with Direct Preference Optimization