When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

2026-08-05

AI Summary

This work addresses the semantic channel gap in multimodal large language models, which struggle to effectively leverage visual text for reasoning about task instructions embedded in images—such as those found in screenshots. The study is the first to explicitly identify and quantify this gap, introducing an end-to-end prompt-region grounding method that operates without OCR or region-level metadata. By embedding task instructions directly into images, the authors construct a Visualized Task Semantics (VTS) benchmark and align question-relevant regions with their semantic meanings through masked image modeling and typed semantic representations, thereby recovering clean visual features from occluded views. Evaluated across four benchmarks, the proposed approach improves VTS accuracy from 58.0 to 66.3 (+8.3 percentage points) while preserving performance on original text-interface tasks.

Abstract

Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels. Yet benchmarks usually place questions in text, leaving it unclear whether models use the same instruction equally well across channels. We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed. Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average. Models often transcribe the visual question correctly yet fail to use it, exposing a semantic channel gap beyond OCR. To reduce this gap, we present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view. At matched training cost, our method raises four-benchmark VTS accuracy from 58.0 to 66.3 while preserving accuracy on the original interface, and requires no OCR or region metadata at inference. Reading task-bearing text and grounding it as an instruction for reasoning are distinct capabilities.

Problem

• multimodal reasoning
• prompt grounding
• visual question understanding
• semantic channel gap
• task instruction in pixels

Innovation

• prompt-region grounding
• Visualized Task Semantics
• multimodal reasoning
• semantic channel gap
• masked representation