Parallelized Autoregressive Visual Generation

2024-12-19

arXiv.org

AI Summary

Autoregressive visual generation suffers from low inference efficiency due to sequential, token-by-token decoding. This paper proposes a dependency-aware parallelization strategy that requires no modification to the model architecture or tokenizer. It dynamically partitions the decoding process into parallel and serial regions based on the conditional dependency strength among visual tokens—marking the first integration of explicit dependency modeling into parallel decoding decisions. The method comprises dependency-aware grouping sampling and localized serial constraints, and is plug-and-play on standard Transformer decoders. Evaluated on image and video generation tasks using ImageNet and UCF-101, it achieves up to 9.5× speedup; critically, it attains 3.6× acceleration without compromising generation quality—outperforming existing parallel decoding approaches. Its core contributions are: (i) zero-modification deployment, (ii) dependency-driven parallelization, and (iii) a general, efficient, and quality-preserving acceleration paradigm for autoregressive visual generation.

Technology Category

Computer Vision: Multi-modal VisionMachine Learning: Deep Generative Models & AutoencodersCognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

User Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Autoregressive models have emerged as a powerful approach for visual generation but suffer from slow inference speed due to their sequential token-by-token prediction process. In this paper, we propose a simple yet effective approach for parallelized autoregressive visual generation that improves generation efficiency while preserving the advantages of autoregressive modeling. Our key insight is that parallel generation depends on visual token dependencies-tokens with weak dependencies can be generated in parallel, while strongly dependent adjacent tokens are difficult to generate together, as their independent sampling may lead to inconsistencies. Based on this observation, we develop a parallel generation strategy that generates distant tokens with weak dependencies in parallel while maintaining sequential generation for strongly dependent local tokens. Our approach can be seamlessly integrated into standard autoregressive models without modifying the architecture or tokenizer. Experiments on ImageNet and UCF-101 demonstrate that our method achieves a 3.6x speedup with comparable quality and up to 9.5x speedup with minimal quality degradation across both image and video generation tasks. We hope this work will inspire future research in efficient visual generation and unified autoregressive modeling. Project page: https://yuqingwang1029.github.io/PAR-project.

Problem

• Slow inference speed in autoregressive visual generation models
• Balancing parallel and sequential token generation efficiently
• Maintaining quality while accelerating image and video generation

Innovation

• Parallel generation of weakly dependent tokens
• Sequential generation for strongly dependent tokens
• Seamless integration into autoregressive models