Sonic: Shifting Focus to Global Audio Perception in Portrait Animation

2024-11-25

arXiv.org

AI Summary

Existing audio-driven portrait animation methods rely on visual priors or local audio modeling, resulting in low motion naturalness and temporal inconsistency. This paper proposes a purely audio-driven paradigm that eliminates all visual guidance, using only raw audio as input. Our method introduces three key innovations: (1) a novel decoupled intra- and inter-utterance audio perception mechanism; (2) a context-enhanced audio encoder, a motion-decoupled controller, and a time-aware positional offset fusion module; and (3) an end-to-end framework integrating long-horizon modeling, audio-motion decoupled control, and sliding temporal window fusion. Quantitative and qualitative evaluations demonstrate state-of-the-art performance across four critical dimensions: lip-sync accuracy, video fidelity, temporal coherence, and motion diversity.

Technology Category

Natural Language Processing: SpeechComputer Vision: Motion & TrackingMachine Learning: Multimodal Learning

Application Category

User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systemsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchResponsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The study of talking face generation mainly explores the intricacies of synchronizing facial movements and crafting visually appealing, temporally-coherent animations. However, due to the limited exploration of global audio perception, current approaches predominantly employ auxiliary visual and spatial knowledge to stabilize the movements, which often results in the deterioration of the naturalness and temporal inconsistencies.Considering the essence of audio-driven animation, the audio signal serves as the ideal and unique priors to adjust facial expressions and lip movements, without resorting to interference of any visual signals. Based on this motivation, we propose a novel paradigm, dubbed as Sonic, to {s}hift f{o}cus on the exploration of global audio per{c}ept{i}o{n}.To effectively leverage global audio knowledge, we disentangle it into intra- and inter-clip audio perception and collaborate with both aspects to enhance overall perception.For the intra-clip audio perception, 1). extbf{Context-enhanced audio learning}, in which long-range intra-clip temporal audio knowledge is extracted to provide facial expression and lip motion priors implicitly expressed as the tone and speed of speech. 2). extbf{Motion-decoupled controller}, in which the motion of the head and expression movement are disentangled and independently controlled by intra-audio clips. Most importantly, for inter-clip audio perception, as a bridge to connect the intra-clips to achieve the global perception, extbf{Time-aware position shift fusion}, in which the global inter-clip audio information is considered and fused for long-audio inference via through consecutively time-aware shifted windows. Extensive experiments demonstrate that the novel audio-driven paradigm outperform existing SOTA methodologies in terms of video quality, temporally consistency, lip synchronization precision, and motion diversity.

Problem

• Enhancing global audio perception in portrait animation
• Reducing reliance on visual signals for facial synchronization
• Improving naturalness and temporal consistency in audio-driven animation

Innovation

• Focuses on global audio perception for animation
• Disentangles audio into intra- and inter-clip perception
• Uses time-aware fusion for long-audio inference