Problem
Research questions and friction points this paper is trying to address.
multimodal reasoning
visual evidence selection
task-conditioned observation
image budgeting
localizable evidence
Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning
🤖 AI Summary
Existing multimodal large language models lack a task-driven mechanism for visual observation, making it difficult to efficiently locate critical visual evidence. This work proposes a query-conditioned visual evidence graph framework that enables coordinate-level observation under a fixed budget on a frozen multimodal reader, explicitly modeling the observation decision process. The approach unifies support for both text-dense figures and natural images without requiring region-level annotations, optimizing observation utility end-to-end via answer correctness. It integrates OCR/layout information, query-conditioned node generation, and budget-constrained observation selection. On V*Bench, the method achieves an accuracy of 0.833 using only 19% of image regions—substantially outperforming the full-image baseline of 0.696—and attains 92% of full-image ANLS performance on InfographicVQA. Comprehensive evaluation across six benchmarks demonstrates its significant advantages in scenarios requiring localized evidence.
📝 Abstract
High-resolution pixels and crop or zoom tools give multimodal large language models the ability to inspect an image, but they do not provide a reliable task-conditioned policy for deciding where to inspect. Q-CueGraph makes this decision explicit. It maps a question and an image representation to budgeted, coordinate-level observations for a frozen reader. Text-rich images use a reusable OCR/layout graph; natural-image search instantiates query-conditioned visual nodes behind the same selection, composition, and budgeting interface. Optional utility refinement learns which candidate crops the frozen reader can use from training-answer correctness, without region-box supervision. With a frozen Qwen2.5-VL-7B reader, Q-CueGraph reaches 0.833 accuracy on V*Bench versus 0.696 for full-image inference from a 19% image-area budget, and reaches 92% of full-image ANLS on InfographicVQA from about half the image area. Across six benchmarks, explicit observation is most valuable when evidence is localizable, the question discriminates its location, and resolution limits full-image reading.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Query-Conditioned Reasoning
Visual Evidence Graph
Budgeted Observation
Frozen Multimodal LLM
OCR/Layout Integration
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
