Motion-Aware Concept Alignment for Consistent Video Editing

2025-06-01

AI Summary

This paper proposes a training-free video semantic editing framework that precisely injects semantic features of a user-provided reference object into designated spatiotemporal regions of a video, while rigorously preserving the original motion trajectories and visual context. Methodologically, it introduces a novel motion-aware concept alignment mechanism, integrating structured noise trajectory manipulation, momentum-based semantic correction, gamma-residual noise stabilization, and latent-space object detection and tracking; additionally, diagonal denoising scheduling and class-agnostic segmentation are incorporated to enhance controllability. Key contributions include: (1) the first CASS (Consistency-Aware Semantic Similarity) metric specifically designed for video semantic editing evaluation; (2) superior performance over state-of-the-art baselines on a newly constructed benchmark dataset, with significant improvements in spatial consistency, motion coherence, and CASS scores; and (3) zero-shot, high-fidelity, temporally consistent video editing without any model training.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Learning to Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce MoCA-Video (Motion-Aware Concept Alignment in Video), a training-free framework bridging the gap between image-domain semantic mixing and video. Given a generated video and a user-provided reference image, MoCA-Video injects the semantic features of the reference image into a specific object within the video, while preserving the original motion and visual context. Our approach leverages a diagonal denoising schedule and class-agnostic segmentation to detect and track objects in the latent space and precisely control the spatial location of the blended objects. To ensure temporal coherence, we incorporate momentum-based semantic corrections and gamma residual noise stabilization for smooth frame transitions. We evaluate MoCA's performance using the standard SSIM, image-level LPIPS, temporal LPIPS, and introduce a novel metric CASS (Conceptual Alignment Shift Score) to evaluate the consistency and effectiveness of the visual shifts between the source prompt and the modified video frames. Using self-constructed dataset, MoCA-Video outperforms current baselines, achieving superior spatial consistency, coherent motion, and a significantly higher CASS score, despite having no training or fine-tuning. MoCA-Video demonstrates that structured manipulation in the diffusion noise trajectory allows for controllable, high-quality video synthesis.

Problem

• Bridging image-domain semantic mixing to video editing
• Injecting reference image features while preserving motion
• Ensuring temporal coherence in video frame transitions

Innovation

• Training-free framework for video semantic mixing
• Diagonal denoising and class-agnostic segmentation
• Momentum-based corrections and noise stabilization