TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation

2026-07-27

AI Summary

This work addresses the challenges of historical information loss and audio-visual desynchronization in real-time long-form digital human video generation under limited cache constraints. To this end, the authors propose an anchor-guided persistent memory framework that maintains appearance consistency through fixed visual anchors and compresses audio-visual sequences into dynamic states. A modality-specific residual attention mechanism is introduced to enable efficient joint generation, while decoupling persistent memory from local denoising dependencies to support cross-chunk parallel inference. Furthermore, the method incorporates reference-aware feature modulation and an anchor-preserving causal context distillation strategy. Experimental results demonstrate that the proposed approach significantly improves appearance consistency and audio-visual synchronization in long video generation while accelerating autoregressive inference.

Abstract

Real-time long-form digital-human generation relies on causal models to extend audio-visual content while preserving subject appearance and audio-video synchronization across successive segments. A bounded cache retains local motion and phonetic context but discards older evidence, whereas attending to the complete generated history is computationally expensive and can propagate accumulated errors. We present \method, an anchor-guided persistent-memory framework for few-step joint audio-video generation. The framework preserves an immutable visual anchor, compresses completed video and audio blocks into fixed-capacity dynamic states, and retrieves those states through modality-specific residual attention without extending the active cache. A reference-aware modulation method additionally conditions video features on dynamic and anchor appearance statistics. Anchor-preserving causal-context distillation varies rollout horizon, prefix provenance, and cache-history reliability while keeping the immutable visual anchor unperturbed. By separating persistent memory from stage-local denoising dependencies, \method further admits stage-parallel execution across blocks, accelerating autoregressive inference without pipeline-specific retraining. We evaluate long-form video continuations with appearance, temporal, synchronization, facial, and speech diagnostics. Results show that \method preserves stable appearance across prompt-conditioned segments and strong audio-visual synchronization under autoregressive generation. Our project page is https://taoliveaigc.github.io/TaoMate.

Problem

• digital human generation
• audio-video synchronization
• real-time generation
• persistent memory
• appearance consistency

Innovation

• anchor-guided memory
• persistent memory
• audio-video synchronization
• stage-parallel generation
• reference-aware modulation