Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

2026-05-31

AI Summary

This work addresses a core challenge in video-level interactive world modeling: enabling user-controllable, real-time, and temporally consistent state evolution. To this end, we present the first systematic survey of the field, identifying three fundamental technical challenges—action-conditioned generation, long-horizon memory, and real-time responsiveness—and introduce a unified evaluation benchmark spanning four key application scenarios: open-world exploration, game engines, autonomous driving, and robotics. We propose a novel paradigm that integrates multimodal scene modeling with standardized evaluation protocols and release an open-source comprehensive resource repository. This effort aims to establish both methodological foundations and community infrastructure to advance the development of next-generation interactive world models.

Abstract

With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI, autonomous driving, etc. Through explicitly incorporating user actions into world state transition, recent literature empowers world modeling with interactivity in an action-conditioned video or 3D generation paradigm, further enhancing controllability over world evolutions and facilitating users to freely traverse, manipulate, navigate, and personalize the state evolution. In this paper, we aim to systematically review recent research trends, technical developments, evaluation benchmarks, and also propose future potential directions in interactive world modeling. Specifically, we first summarize recent efforts and trends in terms of application scenarios, world state evolution, and scene modality. Afterwards, we delve into three crucial technical challenges, including action-conditioned controllability, long-horizon interactions and memory, and action-following responsiveness for real-time interactivity. Furthermore, we also thoroughly compare existing benchmarks and metrics in four specific application fields: open-world exploration, game engine, autonomous driving, and robotics. Finally, we discuss several promising future directions in achieving next-generation interactive world modeling. The corresponding repository is publicly available at: https://github.com/liujiuming123/Awesome-Interactive-World-Model.

Problem

• interactive world modeling
• action-conditioned generation
• video world modeling
• user interaction
• world state transition

Innovation

• interactive world modeling
• action-conditioned generation
• long-horizon interaction
• real-time responsiveness
• benchmark comparison