Twins: Learn to Predict Unified Representations with Focal Loss

2026-07-24

AI Summary

This work addresses the misalignment between semantic features and low-level latent representations in existing continuous multimodal models, which leads to a mismatch in the latent space for understanding and generation tasks. To resolve this, the authors propose Twins, a unified representation framework that constructs a continuous token space by channel-wise concatenation of Vision Transformer (ViT) and Variational Autoencoder (VAE) features. Within a diffusion Transformer, they introduce an improved focal regression loss into the flow-matching process, effectively mitigating the imbalance inherent in jointly optimizing heterogeneous representations. Notably, this is the first application of focal loss to flow matching, significantly enhancing learning efficiency. On ImageNet, the method achieves up to a 10.57 improvement in gFID over MSE loss without requiring classifier-free guidance, demonstrating superior performance in both multimodal understanding and reconstruction fidelity.

Abstract

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.

Problem

• unified multimodal models
• latent space mismatch
• continuous token space
• optimization imbalance
• multimodal understanding and generation

Innovation

• Twins
• unified representation
• focal regression
• flow matching
• multimodal understanding