ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

2026-07-20

AI Summary

Existing methods struggle to jointly reconstruct the 4D dynamics of both the observer and the environment from monocular first-person video, often relying on additional inputs, employing disjoint modeling strategies, and suffering from low inference efficiency. This work proposes ReViV, a unified framework that achieves, for the first time, end-to-end, full-view first-person 4D reconstruction without requiring task-specific priors. Built upon a masked generative egocentric Transformer, ReViV simultaneously models the joint probability distribution of RGB video, camera trajectory, gaze direction, full-body and hand poses, and scene depth. Evaluated on benchmarks such as HoloAssist and HOT3D, ReViV attains state-of-the-art performance across human body, hand, and gaze reconstruction as well as camera tracking, while maintaining efficient and accurate depth estimation.

Abstract

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.

Problem

• egocentric video
• 4D reconstruction
• camera trajectory
• ego-motion
• monocular RGB

Innovation

• 4D reconstruction
• egocentric vision
• multimodal learning
• transformer architecture
• holistic modeling