Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering

2026-07-28

AI Summary

This work investigates whether the understanding and generation branches of unified multimodal models share a transferable semantic space. To this end, the authors propose a cross-branch semantic guidance framework that extracts intervention-based semantic directions from the understanding branch and transfers them to the generation branch for controllable image synthesis. Their experiments reveal, for the first time, a semantic asymmetry between the two branches: the understanding branch encodes object-level semantics, whereas the generation branch relies more heavily on low-level appearance features. The proposed method enables effective semantic transfer from understanding to generation, significantly improving the semantic fidelity of generated images; however, reverse transfer yields limited gains, demonstrating that architectural unification does not inherently guarantee semantic alignment.

Abstract

Unified multimodal models (UMMs) aim to integrate understanding and generation within a single architecture, yet it remains unclear whether these capabilities share a unified and transferable semantic space. This question is fundamentally challenging, as the two branches operate over heterogeneous representations (text tokens vs.\ visual latents) and distinct training objectives, making direct comparison difficult. To address this, we introduce \emph{cross-branch semantic steering}, an intervention-based framework that extracts semantic directions from one branch and applies them to the other. We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness. In contrast, the reverse direction consistently shows limited effectiveness. Our analysis suggests that this asymmetry may be related to a practical representational mismatch: understanding-derived vectors capture transferable, object-centric semantics, while generation-derived vectors primarily encode low-level appearance features. Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.

Problem

• unified multimodal models
• semantic space
• cross-branch steering
• semantic alignment
• multimodal representations

Innovation

• cross-branch semantic steering
• unified multimodal models
• semantic alignment
• controllable image synthesis
• representational asymmetry