Problem
Research questions and friction points this paper is trying to address.
Unified Multimodal Models
Gradient Conflict
Image Understanding
Image Generation
Position-Resolved Interference
Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models
🤖 AI Summary
This study addresses the gradient conflict between image understanding and generation in unified multimodal models. Moving beyond traditional hierarchical resolution diagnostics, it identifies the positional axis as a critical orthogonal factor. We introduce the Position-Resolved Interference Map to quantify gradient conflicts among visual tokens via a single backward pass, and propose a Position-Aware Modulation (PAM) algorithm that precisely removes reverse-alignment components at high-conflict positions, enabling architecture-agnostic gradient intervention. Our analysis reveals that conflicts are most severe within the first quarter of the sequence. Evaluated on Show-o, PAM outperforms hierarchical separation by 21 points on MME and 2.4 points on GenEval while preserving POPE and FID performance, validating the complementarity of decoupling positional and hierarchical factors.
📝 Abstract
Unified multimodal models (UMMs) train image understanding and autoregressive image generation on shared parameters, and the two objectives are known to interfere. Existing diagnoses and remedies operate at the resolution of layers or experts, measuring conflict per layer and resolving it by separating parameters. We argue that this resolution hides an orthogonal axis. Generation in a UMM is next-token prediction over a raster sequence of visual tokens whose roles vary systematically with position, so how strongly a generation gradient interferes with understanding should depend on where in the sequence it originates. We introduce a position-resolved interference map that attributes understanding-generation gradient conflict to visual-token positions within every layer, computed from a single backward pass at $1.2\times$ the cost of a standard backward pass. On Show-o and Janus-Pro, position explains a large share of conflict variance after controlling for depth (partial $η^2=0.31$ vs. $0.35$ for layer on Show-o; $0.15$ vs. $0.30$ on Janus-Pro): the first quarter of the sequence has a mean gradient cosine of $-0.18$ against understanding, the last quarter $-0.02$. The dependence survives per-position gradient-norm normalization, retaining $80%$ of its effect size, and conflict strength tracks semantic content (Spearman $ρ=0.64$). Building on the map, we propose position-aware modulation (PAM), which removes the anti-aligned component of generation gradients only at high-conflict positions without changing the architecture. Under a matched trainable-parameter budget, PAM improves over layer-wise separation by $+21$ MME and $+2.4$ GenEval points on Show-o while matching it on POPE and overall FID; a random-position control recovers about $31%$ of the gain. Position-based and layer-based separation are complementary degrees of freedom and can be combined.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Unified Multimodal Models
Position-Resolved Gradient Conflict
Position-Aware Modulation
Interference Map
Gradient Conflict
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
