DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents

2026-08-01

AI Summary

This work addresses the challenge of predicting 6-degree-of-freedom (6-DoF) object trajectories from minimal inputs, as existing approaches typically rely on privileged modalities such as video sequences, depth maps, or CAD models and lack fine-grained language-action alignment data. The authors propose DreamTraj, a method that directly predicts 6-DoF trajectories from a single RGB image and a task instruction, without generating full videos. DreamTraj is the first to decode trajectories from early denoising intermediate representations of a frozen image-to-video diffusion model, leveraging a lightweight flow-matching Reader, query-key attention, and hidden-state pooling to bypass the error accumulation and computational overhead of conventional generate-then-extract pipelines. Experiments demonstrate that DreamTraj outperforms prior methods—despite their reliance on multi-frame or privileged inputs—in both translation and rotation prediction, while achieving a 4.6× speedup at inference time.

Abstract

Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.

Problem

• 6-DoF trajectory prediction
• object manipulation
• video diffusion models
• language-to-motion alignment
• perception-action loop

Innovation

• 6-DoF trajectory prediction
• video diffusion latents
• language-conditioned motion forecasting
• perception-action loop
• flow-matching decoder