Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision

2026-07-28

AI Summary

This work addresses the frequent violation of fluid physical laws—such as momentum conservation and gravitational effects—in existing video generation models, which often stems from a lack of explicit motion supervision. To tackle this issue, the authors introduce the first fluid dataset combining large-scale Material Point Method (MPM) physics simulations with real-world pouring videos. They propose a lightweight dual-stream image-to-video architecture that builds upon a frozen pretrained diffusion Transformer backbone and incorporates an optical flow decoding branch. By jointly optimizing endpoint error and smoothness losses, the model injects explicit motion priors into the generation process. Evaluated on both 1.3B and 14B parameter scales, the method significantly enhances physical plausibility, achieving up to an 8.75-point improvement on the VideoPhy-2 benchmark, outperforming baselines in human preference studies, and attaining an optical flow error as low as 0.54 pixels.

Abstract

Video diffusion models generate visually compelling content but routinely violate elementary physics when the subject involves fluids: liquid columns break apart in mid-air, container water levels fail to rise as liquid is poured in, and splashes disperse without regard to momentum or gravity. We attribute this gap to the fact that large-scale video-text corpora contain almost no explicit motion supervision, so models learn to imitate fluid appearance rather than dynamics. We address this with two contributions. First, we build a physics-simulation fluid dataset combining 1,638 MPM-simulated pouring/sloshing videos with 2,320 keyword-filtered real pouring videos mined from stock footage, plus two held-out test sets: a 1,515-video real-video benchmark and an 18-prompt text-to-first-frame generalization benchmark. Second, we introduce a dual-stream image-to-video architecture built on a pretrained diffusion-transformer video generator. It augments the standard RGB decoder with a lightweight Optical-Flow Decoder branch trained with explicit end-point-error and smoothness losses, fused into the RGB stream via zero-initialized convolutions so the pretrained backbone starts undisturbed. Only the two decoders are updated; the encoder, temporal transformer, and text encoder remain frozen. Across two model scales (1.3B and 14B) and two test sets, our method improves VideoPhy-2 Physical-Commonsense and Video-Quality scores over the frozen backbone by up to 8.75 and 4.65 points, outperforms a leading open competitor, and is preferred by human raters in a blind study. A direct optical-flow read-out evaluation further shows an end-point error as low as 0.54 pixels in-distribution, confirming the model has internalized a coherent motion prior rather than merely improving surface appearance.

Problem

• fluid video generation
• physics violation
• motion supervision
• video diffusion models
• physical dynamics

Innovation

• physics-grounded video generation
• dual-stream architecture
• optical-flow supervision
• fluid simulation dataset
• diffusion transformer