When Physical Preferences Meet Semantic Constraints: Physical and Semantic Direct Preference Optimization for Text-to-Video Generation

2026-07-18

AI Summary

Text-to-video generation often suffers from a tension between physical plausibility and semantic consistency. This work proposes Physical-Semantic Direct Preference Optimization (PSDPO), the first approach to jointly model both aspects within preference optimization. Built upon the standard DPO framework, PSDPO dynamically adjusts the contribution weights of preference pairs through gradient analysis to mitigate semantic drift and employs a staged optimization protocol to reduce error accumulation—all without introducing additional models or loss terms. Experiments demonstrate that PSDPO achieves up to a 2× improvement in physical plausibility on VideoPhy-2 while maintaining strong semantic consistency on VBench, significantly outperforming existing preference-based methods.

Abstract

Text-to-video (T2V) generation models have achieved strong visual realism, but improving physical plausibility can come at the cost of semantic consistency with the input text. This tension arises because physical preference is typically determined by comparing dynamics between two videos, without accounting for whether either video faithfully depicts the scene specified by the prompt, making physical-semantic conflict a systematic tendency under this supervision paradigm. We formulate this challenge as a constrained preference optimization problem and propose Physical and Semantic Direct Preference Optimization (PSDPO), which modulates each preference pair's contribution based on the agreement between its physical and semantic signals. A gradient-level analysis shows that PSDPO bounds the semantic drift from conflicting pairs to a controllable residual, and further motivates a staged optimization protocol that provably reduces cumulative drift. The resulting method operates entirely within the standard DPO framework, requiring no auxiliary models or additional loss terms. Experiments show that PSDPO improves physical plausibility by up to $2\times$ over the baseline on VideoPhy-2, while maintaining strong semantic consistency on VBench, achieving a more reliable balance than existing preference-based methods.

Problem

• text-to-video generation
• physical plausibility
• semantic consistency
• preference optimization
• physical-semantic conflict

Innovation

• Preference Optimization
• Text-to-Video Generation
• Physical Plausibility
• Semantic Consistency
• Constrained Optimization