VIG-RL: Learning to Search and Insert for Verified Image Grounding

2026-07-30

AI Summary

Existing retrieval-augmented approaches struggle in knowledge-intensive scenarios to dynamically determine when external visual evidence is needed and how to precisely integrate it into multimodal responses, leading to unreliable text-image fusion. This work formulates Verified Image Grounding as an active decision-making process and introduces, for the first time, a reinforcement learning–based, end-to-end trainable ReAct-style agent framework. The proposed method employs a composite reward mechanism to jointly optimize tool invocation and multimodal alignment, thereby dynamically controlling the retrieval and insertion of visual evidence. Evaluated across multiple benchmarks, the approach significantly outperforms static baselines and establishes a new state-of-the-art performance for this task.

Abstract

In knowledge-intensive scenarios, providing reliable interleaved text-image responses requires Verified Image Grounding (VIG): the precise integration of retrieved authentic visual evidence. Existing retrieval-augmented frameworks predominantly rely on decoupled, static pipelines, inherently failing to dynamically reason about when external knowledge is required and where visual assets should be contextually inserted. To bridge this gap, we propose VIG-RL, an autonomous agentic framework that formulates the search-selection-insertion workflow as an active decision-making process. Operating within a dynamic ReAct-style loop, VIG-RL is optimized via reinforcement learning, guided by a composite reward system that holistically evaluates the agent's step-by-step tool execution and final multimodal alignment. Extensive evaluations demonstrate that VIG-RL establishes a new state-of-the-art, significantly outperforming existing static baselines.

Problem

• Verified Image Grounding
• retrieval-augmented generation
• multimodal reasoning
• dynamic insertion
• knowledge-intensive tasks

Innovation

• Verified Image Grounding
• Reinforcement Learning
• Autonomous Agent
• Multimodal Retrieval
• Dynamic Decision-Making