Representation Forcing for Bottleneck-Free Unified Multimodal Models

2026-05-29

AI Summary

This work addresses a key limitation in existing unified multimodal models, which rely on pretrained variational autoencoders (VAEs) for image generation, thereby introducing structural bottlenecks that severely degrade generation quality when removed. To overcome this, the authors propose “Representation Forcing,” a novel approach that redefines visual representations as generative targets rather than perceptual outputs. Specifically, the method autoregressively predicts these representations in pixel space as intermediate tokens and uses them to guide a subsequent pixel-level diffusion process. Notably, this framework operates entirely end-to-end without requiring an external generative latent space, achieving the first bottleneck-free unified multimodal architecture. Experimental results demonstrate that the model matches state-of-the-art VAE-based unified models in image generation quality while consistently outperforming their VAE variants across various image understanding tasks.

Abstract

Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.

Problem

• unified multimodal models
• bottleneck
• image generation
• VAE
• representation learning

Innovation

• Representation Forcing
• Unified Multimodal Models
• Bottleneck-Free
• Pixel-Space Generation
• Autoregressive Representation Prediction