CoT-Edit: Let CoT Guide Instruction Video Editing

2026-08-02

AI Summary

Existing text-driven video editing methods struggle to accurately model spatial relationships and physical constraints in complex scenes, often resulting in ambiguous editing targets and distorted outputs. This work proposes a novel "Plan–Guide–Edit" framework that introduces chain-of-thought reasoning into video editing for the first time. Leveraging a multimodal large language model, the approach performs structured semantic reasoning to generate precise editing instructions annotated with bounding boxes and object attributes, which are then executed by a diffusion model to achieve high-fidelity, spatiotemporally consistent edits. By explicitly bridging semantic intent with spatial execution, the method significantly improves localization accuracy and physical plausibility in multi-object, complex scenarios, surpassing strong baselines with substantially less training data and achieving state-of-the-art performance.

Abstract

Text-driven instruction-based video editing in complex scenes remains challenging: purely textual prompts often fail to capture precise spatial relationships and physical constraints, resulting in target ambiguity and physically implausible outcomes. To address this, we propose a plan--guide--edit framework that explicitly bridges semantic intent and spatial execution. In our framework, a Chain-of-Thought (CoT)-enhanced multimodal large language model (MLLM) serves as a planner, performing structured reasoning over the video and instructions to derive a precise sequence of bounding boxes and attribute-enriched editing directives. These spatial priors then guide a box-conditioned mask generator, transforming ambiguous global retrieval into localized, context-aware refinement and producing masks that more accurately capture object scale, contact relationships, and placement. Building on these spatial and semantic signals, a diffusion-based editor integrates the masks, enriched instructions, and frame features to render high-fidelity edits that remain temporally coherent and spatially well aligned. Trained first in a modular manner and then jointly, our framework achieves superior performance with reduced data requirements, delivering precise localization in scenes with multiple similar objects and physically consistent object additions, and extensive experiments demonstrate state-of-the-art performance over multiple strong baseline methods. More details are available at: https://github.com/flying-sky999/CoT-Edit

Problem

• text-driven video editing
• spatial relationships
• physical constraints
• target ambiguity
• instruction-based editing

Innovation

• Chain-of-Thought
• instruction-based video editing
• spatial reasoning
• diffusion-based editing
• multimodal large language model