RealtimeWAM: One-Step Asynchronous World Action Models

2026-10-05

AI Summary

This work addresses the inference bottlenecks in world action models caused by multi-step denoising and serial expert execution. To overcome these limitations, it proposes a single-step generation and asynchronous inference framework. Methodologically, Teacher-Anchored Consistency Distillation (TACD) is introduced to enable single-step action generation while eliminating iterative error accumulation. Additionally, a Cross-Expert Wavefront Pipeline (CEWP) is designed to facilitate asynchronous inference by overlapping video and action module computations through block-level KV cache sharing. This hybrid architecture incurs less than 1% performance degradation on benchmarks such as LIBERO while achieving an approximately 25× inference speedup on H100 GPUs, substantially enhancing real-time control efficiency.

Abstract

World Action Models (WAMs) incorporate visual representations from video generation backbones to guide action prediction. Recent efficient WAMs adopt Mixture-of-Transformers (MoT) architectures and compute video representations once for reuse by the action expert. However, intra-expert iteration (\ie, multi-step action denoising) and inter-expert waiting (\ie, sequential execution of the video and action experts) still limit inference efficiency. To this end, we present RealtimeWAM, an extremely efficient WAM variant with one-step action generation and asynchronous inference, addressing these two bottlenecks. To reduce intra-expert iteration, we propose Teacher-Anchored Consistency Distillation (TACD) to address a local-global error gap: low local consistency error alone does not guarantee accurate final actions. TACD supplements local consistency with explicit supervision from the frozen teacher's multi-step rollout endpoint, enabling accurate one-step action generation. Additionally, we propose Cross-Expert Wavefront Pipelining (CEWP) to eliminate unnecessary expert-level waiting. It overlaps the two experts through block-wise sharing of the video KV cache, synchronizing only immediately before the corresponding action attention consumes it. Extensive experiments across diverse benchmarks (\eg, LIBERO, LIBERO-Plus and RoboTwin) and model variants (\eg, Fast-WAM and Faster-WAM) demonstrate the superiority of RealtimeWAM. Notably, RealtimeWAM maintains near-lossless performance (\ie, $<1\%$ drop) across these benchmarks while delivering significant end-to-end speedup (\eg, $\sim25\times$ on H100). Our code and checkpoints are available via this \href{https://github.com/ModelTC/LightX2V/tree/main/examples/realtimewam}{link}.

Problem

• World Action Models
• Inference Efficiency
• Multi-step Denoising
• Sequential Execution
• Real-time Inference

Innovation

• World Action Models
• Consistency Distillation
• Asynchronous Inference
• Wavefront Pipelining
• KV Cache Sharing