Flex4DHuman: Flexible Multi-view Video Diffusion for 4D Human Reconstruction

2026-06-11

AI Summary

This work addresses the challenge of efficiently generating dynamic 4D human representations from monocular or sparse multi-view videos without relying on explicit geometric priors such as skeletons or depth maps. The authors propose a diffusion-based multi-view video generation method conditioned solely on relative camera poses, integrating spatiotemporal information and SE(3) camera geometry through a five-axis positional encoding scheme. A three-stage curriculum learning strategy enables flexible novel-view synthesis and temporal extrapolation. Built upon the Wan 2.1 1.3B text-to-video diffusion model, the approach incorporates extended spatiotemporal RoPE, historical target-view tokens, and multi-view textual conditioning. Evaluated on the DNA-Rendering and ActorsHQ datasets, the method outperforms existing approaches, demonstrates generalization across human and animal categories, and enables high-quality dynamic 4D content generation from ordinary monocular videos.

Abstract

We present Flex4DHuman, a multi-view video diffusion model that transforms a monocular or sparse multi-view video of a dynamic subject into synchronized dense multi-view videos using only relative camera-pose conditioning. Unlike prior human-centric methods that rely on skeletons, depth maps, normals, or rendered target-view geometry, Flex4DHuman requires no explicit geometry priors and instead conditions generation through relative camera-pose positional encoding. The generated videos can be directly ingested by downstream reconstruction pipelines to create dynamic 4D Gaussian splats. Built on the Wan 2.1 1.3B text-to-video model, Flex4DHuman preserves the backbone architecture and encodes camera and view information through a five-axis positional encoding that extends spatio-temporal RoPE with view indices and continuous SE(3) relative camera geometry. A three-stage curriculum progressively trains the model for pose following, flexible reference-to-target view generation, and temporal rollout. To support temporal rollout, we train with clean historical target-view tokens. We also add multi-view captions to enable test-time text control. Combined with an off-the-shelf 4D Gaussian Splatting stage, our framework lifts monocular static-camera videos into dynamic 4D Gaussian splats. Experiments on DNA-Rendering and ActorsHQ show that Flex4DHuman surpasses prior state-of-the-art methods, while the same formulation generalizes to animal categories after mixed human-animal training. These capabilities make Flex4DHuman a practical step toward scalable 4D content creation from casual monocular videos for simulation, gaming, AR/VR, and video re-shooting.

Problem

• 4D human reconstruction
• multi-view video generation
• monocular video
• camera-pose conditioning
• dynamic scene modeling

Innovation

• multi-view video diffusion
• relative camera-pose conditioning
• 4D Gaussian splatting
• geometry-free generation
• temporal rollout