VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

2024-09-06

arXiv.org

AI Summary

Traditional vision-language models (VLMs) rely on modular, decoupled architectures and diffusion-based generators, leading to cross-modal misalignment, architectural complexity, and inefficient inference. To address these limitations, VILA-U introduces a unified autoregressive framework that jointly models understanding and generation across video, image, and text modalities. Its key contributions are: (1) a novel unified vision tower enabling precise alignment between discrete visual tokens and textual tokens; (2) empirical validation that a pure autoregressive paradigm achieves image generation quality competitive with state-of-the-art diffusion models; and (3) elimination of dedicated generative modules through joint multimodal pretraining and unified tokenization. Extensive experiments demonstrate that VILA-U achieves state-of-the-art performance on both multimodal understanding and generation benchmarks, while significantly reducing model size and accelerating inference—offering a more scalable and efficient foundation for unified multimodal intelligence.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.

Problem

• Unifies visual understanding and generation tasks
• Eliminates misalignment and complexity in traditional models
• Achieves state-of-the-art performance with simplified architecture

Innovation

• Unified autoregressive framework for visual tasks
• Single model integrates understanding and generation
• Token-based approach matches diffusion model quality