Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization

2026-07-27

AI Summary

This work addresses the limitation of existing automatic prompt optimization methods in multimodal tasks, which often neglect image perception and struggle to diagnose vision-related errors. To overcome this, the authors propose a Cross-Modal Visual Feedback (CMVF) mechanism that, for the first time, incorporates visual information into prompt optimization. CMVF leverages a stronger vision-language model to analyze failure cases, distills task-level visual blind spot patterns, and generates transferable visual checklists through error-aware aggregation to guide prompt rewriting. Notably, this approach incurs no additional inference cost and achieves consistent performance gains—averaging a 2.4-point improvement across 12 VQA datasets and 4 target models, with a maximum gain of 6.5 points. Moreover, the optimized prompts demonstrate strong cross-model transferability.

Abstract

Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results. However, on multimodal tasks, the effectiveness of APO is fundamentally bottlenecked by a blind feedback channel: the optimizer reads the question, the prediction, and the gold answer, but never the input image on which the model failed, and therefore cannot diagnose visually grounded errors. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF). CMVF incorporates (1) a failure-conditioned visual diagnosis stage, in which a stronger optimizer VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns that drive the prompt rewrite. Crucially, the image is consumed only during optimization; the deployed artifact is an ordinary text prompt that runs at the same inference cost as any text-only baseline. Extensive results across 12 VQA datasets and 4 target VLMs demonstrate that CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks. Moreover, the optimizer self-organizes into expert-style visual checklists that transfer across models without re-optimization.

Problem

• automatic prompt optimization
• vision-language models
• visual feedback
• multimodal tasks
• visual error diagnosis

Innovation

• Cross-Modal Visual Feedback
• Automatic Prompt Optimization
• Vision-Language Models
• Visual Diagnosis
• Prompt Engineering