VIPER: Visual In-Context Physics Reasoning for Physically Plausible Video Generation

2026-07-26

AI Summary

Existing video generation models struggle to precisely control complex physical behaviors through text or images, as continuous physical cues—such as material responses, contact interactions, and motion trajectories—are difficult to fully capture with language. This work proposes VIPER, a novel framework that treats reference videos as demonstrations of physical behavior rather than appearance templates. VIPER leverages a multimodal large language model to extract physical cues and employs a hierarchical training strategy to guide a pretrained image-to-video generator, enabling physical behavior transfer while preserving its visual priors. Evaluated on the newly curated VIPER-19K dataset—which includes annotations for materials, trajectories, and physical effects—the method significantly outperforms baselines on unseen validation sets, achieving notable improvements in both physical plausibility and human preference while maintaining high-quality general video generation capabilities.

Abstract

Modern video generation models can synthesize visually compelling and temporally coherent clips, yet controlling their physical behavior remains difficult with standard text and image conditions. The core challenge is a conditioning bottleneck: material response, contact interaction, deformation, and motion trajectory are continuous and relational physical cues that are hard to specify exhaustively in language but can be demonstrated naturally by video. We propose VIPER, a Visual In-Context Physics Reasoning framework for reference-guided image-to-video generation. Given a target image, a brief target prompt, and a reference video, VIPER treats the reference as a dense visual demonstration of the desired physical process rather than an appearance template. It uses a Multimodal Large Language Model (MLLM) to extract reference-derived physical cues and guide a pretrained image-to-video generator through a hierarchical training strategy, enabling physical behavior transfer while preserving the visual prior of the base generator. To support this setting, we construct VIPER-19K, a curated dataset with material, trajectory, and physical-impact annotations, together with filtered reference-target pairs. Experiments on an unseen validation set show that VIPER achieves stronger reference-video physical similarity and higher human preference than representative video generation and video-as-prompt baselines, while maintaining competitive general video quality. Qualitative results further demonstrate that VIPER can transfer reference-derived physical behavior to new target scenes without requiring carefully engineered prompts.

Problem

• physically plausible video generation
• physical behavior control
• visual in-context reasoning
• reference-guided generation
• material and motion dynamics

Innovation

• Visual In-Context Learning
• Physics Reasoning
• Reference-Guided Video Generation
• Multimodal Large Language Model
• Physical Behavior Transfer