FlowAct-R2: Beyond Talking Avatar via Streaming Multimodal References and Proactive Agent Planning

2026-09-28

AI Summary

This study addresses the challenges of discontinuous multimodal control and the absence of proactive planning in interactive human video generation. To this end, it proposes a real-time, long-horizon generation framework that integrates a streaming multimodal reference diffusion Transformer with a proactive interaction agent. Building upon the Seedance 2.0 Mini architecture, the method introduces rolling action prompts and dynamic reference mechanisms to decouple offline planning from online scheduling. Furthermore, it incorporates video-driven rotary position encoding, partially noised historical frames, and multimodal skill reuse to effectively handle interruptions and audience interactions. The proposed framework supports real-time 720p generation and hour-long streaming output while substantially mitigating appearance drift, demonstrating strong applicability to interactive scenarios such as live broadcasting and conversational interfaces.

Abstract

We present FlowAct-R2, a framework for interactive humanoid video generation that combines continuous multimodal control with proactive agent planning. Our method consists of two coupled components. First, a Streaming Multimodal Reference Diffusion Transformer adapts the pretrained Seedance 2.0 Mini reference-to-video backbone to accept rolling action prompts, streaming audio, and dynamically updated image, audio, and video references. Video-driven rotary positional embeddings align reference chunks with the generation timeline, while reference-plus-image conditioning and partially noised historical motion frames preserve appearance and avoid accumulated drift. Second, a Proactive Interaction Agent separates pre-online planning from online scheduling and response: it prepares a persona, a long-horizon agenda, and reusable multimodal skills in advance, then autonomously schedules behaviors, responds to audience input, and handles interruptions during a live session. FlowAct-R2 supports real-time 720p generation and hour-scale streaming across entertainment streaming, live shopping, video chatting, and live vlogging.

Problem

• Interactive Humanoid Video Generation
• Streaming Multimodal Control
• Proactive Agent Planning
• Real-time Video Streaming

Innovation

• Streaming Multimodal Reference Diffusion Transformer
• Proactive Interaction Agent
• Interactive Humanoid Video Generation
• Rotary Positional Embeddings
• Real-time Video Generation