MoRoute: Dynamic Routing for In-Context Multimodal Video Generation

2026-07-31

AI Summary

This work addresses the challenge of effectively integrating heterogeneous pretrained vision-language models (VLMs) with video diffusion Transformers (DiTs) for unified multimodal video generation and editing. The proposed MoRoute framework introduces, for the first time, a lightweight block-level dynamic routing mechanism that enables each DiT layer to adaptively select the most relevant VLM features. By directly embedding reference images and source videos into the DiT token sequence, MoRoute achieves context-aware, unified conditional generation without requiring fine-tuning of either the VLM or the DiT. Evaluated on IntelligentVBench, OpenVE-Bench, and RefVIE-Bench, the method achieves average improvements of 0.15, 0.18, and 0.34 points respectively on a 1–5 scale, significantly outperforming current state-of-the-art approaches.

Abstract

Multimodal video generation aims to generate and edit videos conditioned on arbitrary combinations of text, images, and videos within a single model, allowing diverse tasks to share complementary data and generative priors. Unifying these tasks requires multimodal understanding of diverse conditions, which is typically provided by a pretrained vision-language model (VLM). A key challenge is how to connect the VLM's hierarchical multimodal representations with a pretrained video diffusion transformer (DiT). Existing methods either inject features from only the final or a few manually selected VLM layers, or jointly train architecture-matched understanding and generation streams, making it difficult to reuse heterogeneous pretrained backbones. We introduce MoRoute, a unified multimodal video generation framework that formulates a frozen VLM and a pretrained video DiT with different architectures as heterogeneous experts connected through dynamic layer routing. For each input, a lightweight block-wise router enables every DiT block to select the VLM layer most relevant to its generation stage, thereby learning an adaptive correspondence between multimodal understanding and video synthesis. MoRoute further incorporates reference images and source videos directly into the DiT token sequence through unified in-context conditioning, preserving fine-grained visual details across diverse generation and editing tasks. Experiments on IntelligentVBench, OpenVE-Bench, and RefVIE-Bench show that MoRoute consistently surpasses the best competing method on each benchmark, improving the average score by 0.15, 0.18, and 0.34 on a 1-5 scale, respectively.

Problem

• multimodal video generation
• vision-language model
• video diffusion transformer
• heterogeneous backbones
• dynamic routing

Innovation

• dynamic routing
• multimodal video generation
• in-context conditioning
• heterogeneous experts
• vision-language model