Revising Second Order Terms in Deep Animation Video Coding

2025-10-27

AI Summary

To address the reconstruction artifacts and weak motion modeling capability of the First-Order Motion Model (FOMM) under large head rotations—particularly 3D rotations—this paper proposes an enhanced framework. It replaces the original local Jacobian-based transformation with an explicit global rotation transform to more accurately model head pose dynamics, and incorporates discriminator normalization to stabilize adversarial training and improve generation robustness. The method jointly leverages keypoint-driven motion estimation, image warping, and global rotation modeling, achieving both high visual fidelity and significantly improved compression efficiency. Experiments on videos featuring aggressive head rotations demonstrate a 40–80% reduction in P-frame bit-rate, alongside 12.7% and 9.3% improvements in LPIPS and DISTS metrics, respectively. These results validate the proposed approach’s dual advantages in dynamic motion representation and coding efficiency.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Learning with Manifolds
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

First Order Motion Model is a generative model that animates human heads based on very little motion information derived from keypoints. It is a promising solution for video communication because first it operates at very low bitrate and second its computational complexity is moderate compared to other learning based video codecs. However, it has strong limitations by design. Since it generates facial animations by warping source-images, it fails to recreate videos with strong head movements. This works concentrates on one specific kind of head movements, namely head rotations. We show that replacing the Jacobian transformations in FOMM by a global rotation helps the system to perform better on items with head-rotations while saving 40% to 80% of bitrate on P-frames. Moreover, we apply state-of-the-art normalization techniques to the discriminator to stabilize the adversarial training which is essential for generating visually appealing videos. We evaluate the performance by the learned metics LPIPS and DISTS to show the success our optimizations.

Problem

• Improving facial animation quality during head rotations
• Reducing bitrate consumption for P-frames in video coding
• Stabilizing adversarial training for better video generation

Innovation

• Replaces Jacobian transformations with global rotation
• Applies normalization techniques to stabilize adversarial training
• Saves 40% to 80% bitrate on P-frames