Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

2026-07-23

AI Summary

This work addresses the challenge of maintaining a consistent and continuously evolving shared world state across multiple viewpoints in multi-agent interaction scenarios. To this end, the authors propose WorldWeaver (W²), a streaming multi-agent video diffusion model that introduces, for the first time, a learnable world state register dynamically updated after each generated video segment to jointly preserve global coherence and agent-specific perspectives. W² employs a hybrid Transformer architecture that decouples world state modeling from visual frame generation and integrates multi-view supervision signals—including bird’s-eye-view maps and scene-level textual descriptions. Evaluated on a two-agent Minecraft video generation task, the method demonstrates significant improvements in both logical consistency and visual fidelity.

Abstract

Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings. We present WorldWeaver (W^2), a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. We ground these registers with supervision signals spanning individual agent status, global state views including bird's-eye views, and scene text. We further improve the architecture with a Mixture-of-Transformers design that uses separate weights for world state modeling and visual frame modeling. Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality.

Problem

• multi-agent
• world state
• video diffusion
• autoregressive
• consistency

Innovation

• world state registers
• multi-agent diffusion
• autoregressive video generation
• Mixture-of-Transformers
• streaming world modeling