World Models' Last Exam in Physics

2026-10-06

AI Summary

This study addresses the poor physical consistency of video world models and the limitations of existing evaluations that rely on subjective judgment or reference videos. To this end, it proposes the first reference-free physics benchmark grounded in quantitative measurement. Encompassing 40 tasks spanning mechanics, optics, and other domains, the benchmark integrates observability-based filtering with multi-domain physical simulations to deliver interpretable physical diagnostics through quantitative measurement algorithms and vision-language model comparisons. Experimental results demonstrate that the best-performing model achieves a score of only 57.76, revealing significant and persistent physical inconsistencies in current approaches. Furthermore, the proposed evaluator exhibits superior alignment with human judgment compared to existing baselines.

Abstract

Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems. Existing evaluations often rely on model-based judgments or reference videos, while direct physical tests largely focus on mechanics. We introduce World Models' Last Exam in Physics, a measurement-based benchmark for evaluating physical consistency in video world models. The benchmark comprises 40 controlled tasks spanning mechanics, optics, fluids, thermal and phase-change phenomena, electromagnetism, and surface tension. Each task pairs an initial image and a generation prompt with predefined physical criteria, enabling interpretable tests of observable physical relationships without requiring reference videos. Its evaluator combines task-observability screening with task-specific quantitative physical measurements. Experiments on eight video generation models across 1,280 videos reveal persistent physical inconsistencies and substantial variation across tasks, with the best model achieving an overall score of 57.76 out of 100. Evaluation on synthetic videos with known physical relationships provides evidence for the validity of the measurement module under controlled conditions. The evaluator also achieves higher agreement with human judgments than a direct vision-language model baseline in both within-task rankings and pairwise comparisons. By combining coverage across physical domains with scores grounded in measurable evidence and explicit measurement limitations, the benchmark provides an interpretable basis for diagnosing physical inconsistencies and tracking progress toward physically consistent video world models.

Problem

• Video World Models
• Physical Consistency
• Embodied AI
• Benchmark Evaluation

Innovation

• Video World Models
• Physical Consistency Benchmark
• Quantitative Physical Measurement
• Embodied AI Evaluation
• Measurement-based Evaluator