Text-Driven Video Style Transfer with State-Space Models: Extending StyleMamba for Temporal Coherence

2025-03-15

AI Summary

This paper addresses key challenges in text-driven video style transfer—namely, inter-frame style inconsistency, motion jitter, and poor robustness under scene transitions. To this end, we propose a temporal-aware approach grounded in State Space Models (SSMs). Our method introduces three core innovations: (1) a Video State Space Fusion module that jointly models spatiotemporal features across frames; (2) a Temporal Masked Directional Loss to enforce style consistency in dynamically occluded regions; and (3) a Temporal Second-Order Loss that regularizes the second-order differences of optical flow to enhance motion smoothness. Evaluated on DAVIS and UCF101, our method achieves significant improvements in style consistency (+12.6% FID) and visual temporal coherence. Moreover, it outperforms mainstream diffusion- and Transformer-based baselines in computational efficiency, enabling near-real-time video stylization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

StyleMamba has recently demonstrated efficient text-driven image style transfer by leveraging state-space models (SSMs) and masked directional losses. In this paper, we extend the StyleMamba framework to handle video sequences. We propose new temporal modules, including a emph{Video State-Space Fusion Module} to model inter-frame dependencies and a novel emph{Temporal Masked Directional Loss} that ensures style consistency while addressing scene changes and partial occlusions. Additionally, we introduce a emph{Temporal Second-Order Loss} to suppress abrupt style variations across consecutive frames. Our experiments on DAVIS and UCF101 show that the proposed approach outperforms competing methods in terms of style consistency, smoothness, and computational efficiency. We believe our new framework paves the way for real-time text-driven video stylization with state-of-the-art perceptual results.

Problem

• Extend StyleMamba for text-driven video style transfer
• Ensure temporal coherence in video style transfer
• Improve style consistency and smoothness in video sequences

Innovation

• Video State-Space Fusion Module for inter-frame dependencies
• Temporal Masked Directional Loss for style consistency
• Temporal Second-Order Loss to suppress abrupt variations