StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

2026-07-22

AI Summary

This work addresses the challenge of simultaneously achieving low latency for real-time interaction and long-term temporal coherence in human-object interaction (HOI) video generation. We propose the first streaming HOI video generation framework designed for low latency, which analyzes the historical memory preferences of individual Transformer blocks to devise an interaction-aware memory organization mechanism. This mechanism introduces block-specific memory layouts and a long-range interaction state access strategy, overcoming the limitations of conventional sink-local memory constraints. Furthermore, we integrate bias-guided training and a memory distance scaling module to enable efficient generation. Experiments demonstrate that our method significantly outperforms existing approaches in interaction plausibility, object fidelity, human quality, and efficiency, achieving 17.6 FPS with a first-block latency of only 0.75 seconds.

Abstract

Existing human--object interaction (HOI) video generation methods are largely limited to offline short-video generation with complex driving conditions, making them unsuitable for real-time interactive applications. We present \emph{StreamHOI}, a low-latency streaming framework for long-duration HOI video generation. Instead of converting heavily conditioned HOI pipelines into streaming systems, we study how an image-to-video streaming generator should organize historical memory to preserve interactions under bounded latency. We find that the standard sink-local memory design faces a trade-off in streaming HOI generation, and different transformer blocks show different historical-memory preferences for HOI regions and surrounding regions. To match memory composition with block behavior, StreamHOI performs offline HOI-aware block profiling and applies bias-guided memory-specialized training to adapt the generator to block-specific memory layouts. We further introduce a memory distance scaling module to strengthen long-range access to early interaction states. Extensive comparisons with both long-video baselines and recent HOI generation methods demonstrate that StreamHOI achieves strong interaction plausibility, object fidelity, human quality and efficiency, reaching 17.6 FPS with 0.75s first-chunk latency.

Problem

• Human-Object Interaction
• Streaming Video Generation
• Temporal Memory
• Real-time Generation
• Long-duration Video

Innovation

• streaming video generation
• human-object interaction
• temporal memory adaptation
• memory-specialized training
• long-range memory access