DepthWorld: 3D World Model for Robot Manipulation

2026-10-06

AI Summary

This study addresses the lack of 3D geometric consistency in existing video world models, which hinders their applicability to robotic manipulation requiring precise spatial understanding. We propose the DROID-3D dataset and a depth-aware world model. Specifically, we construct a large-scale 3D-supervised calibration pipeline leveraging stereo vision and factor graph optimization, and design a spatial latent tiling architecture that jointly predicts multi-view RGB images and metric depth while preserving pretrained priors. Our approach achieves a reprojection error below 0.7 pixels, improves RGB prediction PSNR by 1.48 dB, and generates high-fidelity depth maps that facilitate downstream tasks. This work establishes a new paradigm for geometrically consistent video generation in embodied intelligence.

Abstract

World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

Problem

• world models
• robot manipulation
• 3D geometry
• video prediction
• depth supervision

Innovation

• 3D World Model
• Depth Supervision
• Spatial Latent Tiling
• Calibration Pipeline
• Robot Manipulation