Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

2026-07-16

AI Summary

This work addresses the “question-first paradox” in vision-language models (VLMs), where placing the question before the image—though intuitively sound—degrades performance. To resolve this conflict between perceptual guidance and question accessibility, the authors propose “Prompt Echo,” a training-free strategy that repeats the question both before and after the image: the leading instance steers visual perception, while the trailing one is used during answer generation. Inspired by repetition effects in human reading, this approach is the first to explicitly identify and mitigate the paradox. The mechanism is validated through logit-lens analysis, attention probing, and causal ablation studies, leading to the design of a bilateral prompting architecture. Evaluated on benchmarks including NaturalBench, POPE, Winoground, and VQAv2, Prompt Echo consistently outperforms optimal single-prompt ordering, achieving up to a 19-percentage-point gain in group accuracy on Winoground.

Abstract

Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace the paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show the intuition is half right: a question placed before the image genuinely steers perception, moving image patch representations toward question-relevant concepts. The failure lies downstream. Stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven (often wrong) answers; a causal attention knockout confirms that the answer reads the question only when the question follows the image. The diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so that one copy steers perception while the other is read out at answer time. The same division of labor appears in a fifty-year-old finding on human ``adjunct questions'', where repeating a question before and after a passage aids comprehension more than either position alone. Echoing the image as well brings further gains, restoring the whole-image view a causal decoder otherwise loses. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts close it and surpass the best single-pass ordering on NaturalBench, POPE, Winoground, and open-ended VQAv2, by up to 19 Winoground group-accuracy points, with no training, fine-tuning, or architecture change. The paradox reveals a trade-off between steering perception and preserving question access; echoing resolves it through prompt design alone.

Problem

• vision-language models
• prompt ordering
• question-first paradox
• visual question answering
• attention mechanism

Innovation

• question echoing
• vision-language models
• prompt design
• causal attention
• question-first paradox