PE-Field 4D: Video Generation Models as Canvas

2026-07-17

AI Summary

This work addresses the challenge of spatial control in video generation arising from viewpoint changes and camera motion by proposing a geometry-aware diffusion Transformer architecture. By integrating projected positional encoding and a depth-aware disambiguation mechanism, the method effectively fuses 3D depth information with 2D reprojection. It further introduces structured context tokens and geometry-guided cross-attention to enable precise spatial manipulation directly within the native latent space. The proposed approach significantly enhances controllability for viewpoint-dependent editing tasks, supporting camera trajectory redirection, novel view synthesis, and geometry-consistent video editing, all while preserving the strong generative priors of the underlying foundation model.

Abstract

Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.

Problem

• video generation
• scene geometry
• viewpoint changes
• camera motion
• spatial controllability

Innovation

• positional encoding
• geometry-aware control
• video diffusion transformers
• cross-attention mechanism
• novel-view synthesis