DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation

2026-08-03

AI Summary

This work addresses the computational inefficiency of existing video prediction methods, which often rely on pixel-level generation or heavy decoders. The authors propose DF³, a novel framework that introduces learnable spatial queries at the terminal layer of a frozen vision foundation model. By leveraging a motion-aware context fusion (MACF) mechanism, DF³ directly predicts future features in latent space and employs task queries to output navigation decisions—entirely eliminating the need for a decoder. This approach establishes the first end-to-end world model that operates without a decoder, unifying state prediction and task execution within the latent space. Evaluated on public benchmarks and zero-shot robotic simulation deployments, DF³ achieves state-of-the-art performance while significantly enhancing computational efficiency and perception-action integration.

Abstract

Forecasting future states from video sequences is a critical challenge for autonomous robotic systems and a fundamental objective of world modeling. Prior generative methods operating at the pixel level inevitably overemphasize task-irrelevant details, leading to prohibitive computational overhead. While latent-based approaches attempt to mitigate this by predicting features directly, the persistent reliance on heavy decoders for state-to-task mapping remains a computational bottleneck. In this work, we propose Decoder-Free Feature Forecasting (DF$^3$), a novel framework that models world evolution entirely within the latent space and directly derives task outputs, completely eliminating the need for a decoder. Specifically, DF$^3$ injects learnable spatial queries into the terminal blocks of a frozen vision foundation model to extract future state representations directly. By employing a lightweight, unified Motion-Aware Context Fusion (MACF) mechanism that seamlessly integrates coarse flow warping with fine-grained latent cross-correlation, these queries interact with historical token representations to explicitly align and forecast the feature of the next frame. Subsequently, a specialized set of task queries probes these forecasted features for the downstream task. Extensive experiments on public benchmarks and zero-shot deployment in a robotic simulator demonstrate that DF$^3$ achieves performance comparable to state-of-the-art methods while offering superior efficiency and flexibility for integrated perception and control.

Problem

• world modeling
• feature forecasting
• autonomous navigation
• decoder-free
• latent space

Innovation

• Decoder-Free
• Feature Forecasting
• Latent Space Modeling
• Motion-Aware Context Fusion
• World Modeling