TranStable: Towards Robust Pixel-level Online Video Stabilization by Jointing Transformer and CNN

2025-01-25

AI Summary

To address severe geometric distortion and excessive cropping in video stabilization, this paper proposes TranStable, an end-to-end framework. Methodologically, it introduces a novel TransformerUNet (TUNet) generator and a Stability Discrimination Module (SDM), integrated via a Hierarchical Adaptive Fusion Mechanism (HAFM) to jointly model global motion consistency and preserve local structural fidelity. Furthermore, it employs pixel-wise warping map regression coupled with adversarial training under realism-aware supervision. Evaluated on NUS, DeepStab, and Selfie benchmarks, TranStable achieves state-of-the-art performance: it significantly suppresses jitter artifacts, enhances field-of-view retention and visual fidelity, and effectively alleviates the distortion–cropping trade-off inherent in conventional approaches.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Video stabilization often struggles with distortion and excessive cropping. This paper proposes a novel end-to-end framework, named TranStable, to address these challenges, comprising a genera tor and a discriminator. We establish TransformerUNet (TUNet) as the generator to utilize the Hierarchical Adaptive Fusion Module (HAFM), integrating Transformer and CNN to leverage both global and local features across multiple visual cues. By modeling frame-wise relationships, it generates robust pixel-level warping maps for stable geometric transformations. Furthermore, we design the Stability Discriminator Module (SDM), which provides pixel-wise supervision for authenticity and consistency in training period, ensuring more complete field-of-view while minimizing jitter artifacts and enhancing visual fidelity. Extensive experiments on NUS, DeepStab, and Selfie benchmarks demonstrate state-of-the-art performance.

Problem

• Video Stabilization
• Distortion
• Cropping

Innovation

• TransformerUNet
• Stability Inspection Module
• Video Stabilization