Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

2026-07-28

AI Summary

This work addresses the challenges of unified image understanding and generation—namely high computational costs, substantial data requirements, and conflicting task-specific characteristics—by proposing a lightweight unified framework. The approach freezes the vision encoder of a pretrained vision-language model and introduces a hybrid visual token mechanism: continuous tokens for understanding tasks and discrete tokens for generation. A two-stage training strategy optimizes the quantizer, image decoder, and language model. Remarkably, with only 15.6 million training samples and an estimated cost of approximately \$2,000, the model achieves state-of-the-art performance on GQA, POPE, and VQAv2 for understanding while matching the generation quality of specialized models. This represents a tenfold reduction in cost and a fivefold decrease in data compared to existing methods.

Abstract

Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a frozen unified vision encoder. Our training pipeline includes two stages: the first stage learns a quantizer and image decoder on top of the frozen vision encoder, the second stage trains the LLM initialized from a pretrained VLM for the unified multimodal modeling. Using by far the least amount of data (15.6M) and the lowest cost (~$2,000), we demonstrate that unified multimodal models can be trained economically while achieving strong performance in both understanding and generation. Notably, our model attains state-of-the-art multimodal understanding on GQA, POPE, and VQAv2, and competitive generation quality compared to models with dedicated vision encoders (e.g., Janus, Janus-Pro), all at ~10x lower cost and with ~5x less data. We envision Argus-Unified as a useful baseline that lowers the development barrier for unified models.

Problem

• unified model
• image understanding
• image generation
• multimodal learning
• computational efficiency

Innovation

• unified multimodal model
• hybrid visual tokens
• pretrained vision-language models
• efficient training
• discrete-continuous tokenization