CART: Compositional Auto-Regressive Transformer for Image Generation

2024-11-15

arXiv.org

AI Summary

Traditional autoregressive models for image generation suffer from difficulties in modeling spatial dependencies, compromising semantic interpretability, resolution scalability, and generation controllability. To address this, we propose the Compositional Autoregressive Transformer (CAR-Transformer), which decomposes an image into a base map and multi-level detail factors. Generation proceeds hierarchically and iteratively via fine-grained incremental prediction, departing from conventional token-wise or scale-wise modeling paradigms. CAR-Transformer introduces the novel “compositional autoregression” paradigm, enabling zero-shot resolution scaling—arbitrary high-resolution images can be synthesized without retraining. While maintaining computational efficiency, the model achieves state-of-the-art performance on high-fidelity image synthesis benchmarks, with significant improvements in generation controllability and cross-resolution generalization.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Deep Generative Models & AutoencodersNatural Language Processing: Generation

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In recent years, image synthesis has achieved remarkable advancements, enabling diverse applications in content creation, virtual reality, and beyond. We introduce a novel approach to image generation using Auto-Regressive (AR) modeling, which leverages a next-detail prediction strategy for enhanced fidelity and scalability. While AR models have achieved transformative success in language modeling, replicating this success in vision tasks has presented unique challenges due to the inherent spatial dependencies in images. Our proposed method addresses these challenges by iteratively adding finer details to an image compositionally, constructing it as a hierarchical combination of base and detail image factors. This strategy is shown to be more effective than the conventional next-token prediction and even surpasses the state-of-the-art next-scale prediction approaches. A key advantage of this method is its scalability to higher resolutions without requiring full model retraining, making it a versatile solution for high-resolution image generation.

Problem

• Models images as hierarchical compositions of interpretable visual layers
• Addresses spatial dependency challenges in autoregressive image generation
• Enables controllable image manipulation through semantic decomposition strategies

Innovation

• Models images as hierarchical compositions of interpretable layers
• Iteratively adds details via semantic decomposition strategies
• Outperforms traditional approaches with improved controllability and scalability