UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling

2026-07-27

AI Summary

Current visual generation tasks rely on specialized models, lacking a unified and efficient framework, and diffusion models suffer from high inference latency. This work proposes UniGen-AR, the first unified generative architecture based on Visual Autoregressive (VAR) modeling, which integrates a multimodal language model, a hierarchical VQ-VAE, and a unified sequential control interface to support over 15 image generation tasks. The study reveals that the design of the VQ-VAE tokenizer critically influences VAR scalability. UniGen-AR matches or surpasses baseline generation quality across four major task categories while achieving up to a 19× reduction in inference latency compared to diffusion-based methods, demonstrating the efficiency and feasibility of VAR for unified visual generation.

Abstract

Modern computer vision pipelines remain fragmented, with tasks such as text-to-image generation, editing, restoration, and classical perception handled by separate models. We study Unified Visual Generation (UVG), where a single model produces diverse image-valued outputs through a unified multimodal interface. While diffusion-based systems dominate UVG due to strong quality and controllability, their iterative sampling incurs substantial inference latency, limiting practical deployment. To address these limitations, we propose UniGen-AR, a framework that pairs a general-purpose multi-modal language model (MLLM) with an efficient next-scale visual auto-regressive (VAR) decoder. This design retains the flexibility of MLLM-based conditioning while leveraging the sampling efficiency and latent unification properties of VAR models. In our framework, the MLLM encodes free-form instructions and control signals into a unified sequence, which guides the VAR decoder to generate image-valued outputs for over 15 tasks spanning four families. Empirically, UniGen-AR achieves up to $19 \times$ lower inference latency than diffusion-based baselines while maintaining or improving output quality. Our ablations further reveal that VQ-VAE tokenizer design, particularly codebook size and hierarchy, is a critical factor for VAR scalability in UVG. These results establish visual auto-regressive modeling as a compelling and efficient backbone for unified visual generation. Our project page is at https://zpbao.github.io/projects/unigenar.

Problem

• Unified Visual Generation
• Auto-Regressive Modeling
• Inference Latency
• Multimodal Interface
• Visual Generation

Innovation

• Unified Visual Generation
• Auto-Regressive Modeling
• Multi-Modal Language Model
• VAR Decoder
• VQ-VAE Tokenizer