4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

2026-10-02

AI Summary

This study addresses the difficulty existing AI agents face in reconstructing complex physical behaviors of dynamic scenes from videos. To this end, we propose 4DCodeBench, a novel evaluation framework for 4D inverse graphics based on code generation, which requires agents to translate videos into executable graphics programs that reconstruct dynamic scenes. This framework integrates video understanding, code generation, and physical simulation, encompassing complex physical phenomena such as deformation and fluid dynamics. Experiments reveal a significant performance gap between state-of-the-art models in static reconstruction versus dynamic physical understanding. Consequently, 4DCodeBench provides a new testbed for evaluating the capacity of advanced models to interpret world dynamics.

Abstract

We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as executable graphics programs. To accomplish this, agents must translate visual observations into compact representations of scene structure and dynamics, by implementing abstractions such as physical simulations to reproduce complex behavior. To evaluate this capability, we curate a set of real-world videos and construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow, and fracture. We perform extensive benchmarking of frontier models, finding that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward agents that can interpret the dynamics of the world through code. Our benchmark is available at https://github.com/4DCodeBench/4DCodeBench

Problem

• 4D inverse graphics
• code generation
• dynamic scene reconstruction
• benchmarking
• physical simulation

Innovation

• 4D inverse graphics
• code generation
• dynamic scene reconstruction
• benchmark
• physical simulation