CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects

2024-01-18

arXiv.org

AI Summary

To address the challenges of identity preservation and weak semantic consistency in personalized text-to-video generation for multi-subject scenarios, this paper proposes the first text-to-video framework supporting collaborative multi-subject guidance. Methodologically, we introduce a novel co-occurring multi-subject image construction strategy, design a subject-decoupled attention mechanism in the diffusion model’s latent space, and propose an object-mask-guided attention learning paradigm. Based on this framework, we establish MultiSubjectVid—the first open-source benchmark dataset for multi-subject text-to-video generation. Extensive experiments demonstrate that our approach achieves state-of-the-art performance across identity fidelity, temporal consistency, and text-video alignment accuracy. Both quantitative evaluations and qualitative analyses—complemented by user studies—confirm significant improvements over prior methods. This work establishes a new paradigm for controllable, multi-subject video generation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Customized text-to-video generation aims to generate high-quality videos guided by text prompts and subject references. Current approaches for personalizing text-to-video generation suffer from tackling multiple subjects, which is a more challenging and practical scenario. In this work, our aim is to promote multi-subject guided text-to-video customization. We propose CustomVideo, a novel framework that can generate identity-preserving videos with the guidance of multiple subjects. To be specific, firstly, we encourage the co-occurrence of multiple subjects via composing them in a single image. Further, upon a basic text-to-video diffusion model, we design a simple yet effective attention control strategy to disentangle different subjects in the latent space of diffusion model. Moreover, to help the model focus on the specific area of the object, we segment the object from given reference images and provide a corresponding object mask for attention learning. Also, we collect a multi-subject text-to-video generation dataset as a comprehensive benchmark, with 63 individual subjects from 13 different categories and 68 meaningful pairs. Extensive qualitative, quantitative, and user study results demonstrate the superiority of our method compared to previous state-of-the-art approaches. The project page is https://kyfafyd.wang/projects/customvideo.

Problem

• Customizing text-to-video generation with multiple subject references
• Solving identity preservation challenges in multi-subject video synthesis
• Disentangling multiple subjects in diffusion model latent space

Innovation

• Composes multiple subjects in single image
• Uses attention control in diffusion model
• Segments objects with masks for learning