The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models

2026-07-25

AI Summary

This work addresses the issue in frozen vision-language models where learnable gating mechanisms are inadvertently disabled by optimizers during training, thereby collapsing auxiliary signal pathways. The authors identify gradient vanishing and negative utility as the underlying causes and propose a fixed-scale injection strategy that eliminates the need for learnable gates. Innovatively integrating entailment cones and angular repulsion on hyperbolic manifolds into LoRA fine-tuning, they regularize the model with a geometric auxiliary loss. During inference, the geometric pathway is retained, preserving relational question-answering accuracy while enhancing attribute-based performance. Notably, on out-of-distribution Visual Spatial Relation (VSR) tasks, the geometric loss stabilizes spatial signals—its removal leads to a 4.6 percentage point drop in performance.

Abstract

Auxiliary signal pathways in VLMs are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit. We find that the optimiser almost always decides on zero: across five injection designs, every gated pathway becomes behaviourally closed, with accuracy invariant to ablating the pathway at inference even when the gate parameter would nominally pass 30-45% of the signal. We attribute this suppression phenomenon to two regimes, a dead-gradient regime formalised through the caption-invariance of image-derived signals, and a negative-utility regime in which the auxiliary signal actively hurts the loss. Rather than fight suppression, we exploit it: we regularise LoRA fine-tuning with geometric auxiliary losses from hyperbolic visual relational graphs (IoA-driven entailment cones and angular repulsion on the Lorentz manifold), coupled only through the forward pass at training time and dropped at inference. Disaggregating GQA by question type exposes a clean dissociation. Three configurations without geometric losses at inference lose 2.85-3.39pp on relational questions while gaining ~1pp on attribute questions; a fourth that trains with the losses but infers through a soft prompt loses 5.14pp on rel for only +0.23pp on attr, so training-time regularisation alone does not protect relational accuracy without a geometric inference pathway. Configurations that keep the geometric pathway at inference preserve vanilla-level relational accuracy and match the attribute gain. Out of distribution on VSR, the RMS-prefix recipe preserves the spatial signal; stripping the geometric losses (G2) collapses VSR by 4.6pp, isolating them as the OOD source. A secondary result: embedding-norm alignment is necessary for generation-safe prefix injection, and learnable gates should be replaced with fixed, non-optional injection at matched scales.

Problem

• vision-language models
• auxiliary signals
• gated pathways
• relational reasoning
• out-of-distribution generalization

Innovation

• frozen vision-language models
• auxiliary signal suppression
• geometric regularization
• hyperbolic relational graphs
• LoRA fine-tuning