TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation

2026-06-06

AI Summary

Existing video generation and editing approaches often rely on task-specific models, limiting their ability to handle diverse visual conditions and tasks within a unified framework. This work proposes the first diffusion Transformer-based unified architecture for video synthesis and manipulation. The method introduces token-level task embeddings to isolate distinct tasks, designs a dual-path conditioning mechanism to effectively fuse semantic and structural information, and employs a progressive multi-task training strategy. Evaluated across multiple video generation and editing benchmarks, the proposed approach significantly outperforms current state-of-the-art methods and demonstrates, for the first time, that a single model can efficiently support a wide range of video editing and generation tasks.

Abstract

Recent advances in Diffusion Transformers have driven rapid progress in video generation and editing, yet these capabilities are still handled by separate, task-specific models. Building a unified framework that supports diverse video tasks remains an open challenge: existing unified attempts either require dedicated auxiliary encoders or lack explicit mechanisms to distinguish heterogeneous conditioning tokens, struggling when the number and type of visual conditions vary across tasks. We propose TIDE, a unified framework that integrates instruction-based editing, reference-guided editing, and multi-reference generation. At its core, we introduce per-token task embeddings that assign each input token a task-specific identifier, enabling the model to explicitly disambiguate target, source, and reference tokens. To simultaneously capture high-level semantic understanding and fine-grained structural fidelity, we design a dual-path conditioning scheme that couples a vision-language model with a VAE latent path for complementary signals. We further devise a multi-task progressive training strategy that incrementally introduces tasks of increasing complexity, effectively harmonizing diverse objectives and enabling smooth generalization across heterogeneous task distributions. Extensive experiments on multiple video editing and generation benchmarks demonstrate that TIDE achieves state-of-the-art performance across all evaluated tasks. Our project page is available at https://LittleWork123.github.io/tide.

Problem

• unified video editing
• video generation
• heterogeneous conditioning
• task-specific models
• diffusion transformers

Innovation

• Task-Isolated Diffusion
• Per-token Task Embeddings
• Dual-path Conditioning
• Unified Video Editing and Generation
• Progressive Multi-task Training