WorldDiT: A Unified Diffusion Architecture for World and Action Modeling

2026-07-26

AI Summary

This work addresses the high computational cost and large parameter count associated with existing robotic policies that rely on large pretrained vision-language models (VLMs). The authors propose a unified diffusion Transformer architecture that, for the first time, jointly models continuous action generation and block-level prediction of future RGB image frames within a single framework, eliminating the need for external VLMs. By integrating action and visual world modeling into a cohesive learning objective, the method achieves state-of-the-art performance on four LIBERO simulation benchmarks, establishing a strong baseline for models under one billion parameters and attaining the Pareto frontier in terms of both model size and average success rate.

Abstract

Many recent robot policies pursue stronger control by using large pretrained vision-language models (VLMs) as the action backbone. We introduce WorldDiT, a unified diffusion transformer architecture that couples action generation with visual world modeling and achieves strong performance without a large pretrained VLM action backbone. During training, a single diffusion transformer generates continuous action chunks and predicts normalized RGB patch targets from future camera frames. Across four LIBERO simulation suites, WorldDiT lies on the reported Pareto frontier for total model parameters and mean success among methods reporting all four suites. These results provide a strong sub-billion-parameter baseline for future scaling studies.

Problem

• robot policy
• action generation
• visual world modeling
• diffusion architecture
• pretrained VLM

Innovation

• WorldDiT
• diffusion transformer
• world modeling
• action generation
• robot policy