Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh

2026-07-30

AI Summary

This work addresses the challenge of accurately synchronizing camera motion with scene animation in 4D generation using pretrained video diffusion models, given dynamic meshes, camera trajectories, and reference images. The authors propose DAR, a novel approach that constructs a neural 4D G-buffer—encoding world-coordinate positions and normals—by jointly leveraging pixel-level tracking and 3D geometry, thereby replacing conventional depth maps to disentangle camera and object motion. This G-buffer is integrated into the Wan2.2 architecture via an extended control adapter, enabling high-fidelity 4D rendering while preserving pretrained priors. Combining LoRA and full-parameter fine-tuning strategies, DAR significantly outperforms existing methods on the newly introduced DAR-4D benchmark, achieving PSNR 23.22, SSIM 0.895, and LPIPS 0.134 with LoRA, and further improving to PSNR 25.36 and SSIM 0.917 with full fine-tuning.

Abstract

Pretrained video diffusion models can act as renderers when the desired scene state is already specified by an animated mesh, a camera trajectory, and a reference image. This 4D generative rendering setting raises a representation question: what image-format condition lets a video backbone obey both camera motion and scene-internal animation? We propose DAR, a reference-guided renderer that extends Wan2.2 camera control from Plücker rays alone to a joint camera-plus-geometry interface. DAR projects a neural 4D G-buffer (tracking, world position, and normal) from the animated mesh and injects it through a widened control adapter while preserving the pretrained image-to-video prior. The central design choice is the pair of tracking and world position. Tracking identifies the persistent surface element that should carry appearance; world position gives its current scene-coordinate state; normal supplies local shape. Depth plus calibrated rays can recover 3D in principle, but depth is a camera-dependent chart in which camera and object motion are mixed. On the 68-case DAR-4D benchmark, LoRA DAR reaches PSNR 23.22, SSIM 0.895, and LPIPS 0.134, improving over off-the-shelf Wan2.2-Depth by 1.54 dB PSNR; a full fine-tune reaches PSNR 25.36 and SSIM 0.917. Matched ablations show that replacing world position by depth reduces PSNR by 1.26--1.55 dB at every checkpoint, supporting tracking+world-position correspondence as a practical 4D rendering condition.

Problem

• 4D rendering
• video diffusion models
• animated mesh
• camera motion
• scene animation

Innovation

• 4D rendering
• video diffusion models
• animated mesh
• G-buffer conditioning
• camera-geometry interface