Visual prompt engineering for video models

2026-07-28

AI Summary

This work addresses the limited performance of existing video foundation models on visual reasoning tasks, which stems primarily from the absence of effective prompting mechanisms to unlock their full potential. The study introduces and systematically validates Visual Prompt Engineering (VIPE)—a novel paradigm that leverages image-editing models to automatically transform inputs such as sketches into diverse, photorealistic visual prompts, which are then used as preprocessed inputs to video models. Empirical results demonstrate that VIPE consistently outperforms conventional text-based prompt engineering and test-time scaling strategies across multiple visual reasoning benchmarks. These findings establish VIPE as an efficient, low-compute model enhancement approach that significantly boosts performance without requiring architectural modifications or additional training.

Abstract

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page at https://visual-prompt-engineering.github.io/.

Problem

• visual prompt engineering
• video models
• foundation models
• visual reasoning
• prompt engineering

Innovation

• visual prompt engineering
• video foundation models
• visual reasoning
• image editing
• prompt engineering