Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations

2026-08-01

AI Summary

Existing vision-language models struggle to capture the interaction between images and text in terms of local geometric structure and often overlook their intrinsic representation characteristics—globally high-dimensional yet locally low-dimensional. This work proposes LENS, a novel approach that, for the first time, treats local geometric neighborhoods as interpretable and causal units. Leveraging a Mixture of Factor Analyzers (MFA), LENS decomposes model activations into locally low-rank Gaussian subspaces, thereby uncovering deep dependency trajectories underlying multimodal fusion. By integrating semantic annotations with activation interpolation, LENS enables targeted cross-modal interventions. Experiments on LLaVA and Qwen3-VL demonstrate substantial improvements in controllable generation and retrieval: MFA outperforms baselines such as VL-SAE by up to 5.7× in intervention tasks, and image-to-text retrieval R@1 on Qwen3-VL rises from 14.9% to 48.6%.

Abstract

Vision-language models (VLMs) process image patches and text tokens in a shared residual stream, but the local geometry through which the two modalities interact remains poorly understood. Most interpretability methods identify global linear directions, which may miss representations that are globally high-dimensional but locally low-dimensional. We introduce LENS (Local Explanation of Neighborhood Subspaces), a method that decomposes VLM activations into local low-rank Gaussian neighborhoods using a Mixture of Factor Analyzers. Applied to LLaVA-1.5-7B and Qwen3-VL-8B, LENS reveals distinct depth-dependent fusion trajectories consistent with each model's fusion mechanism: LLaVA progressively mixes modalities at later layers, whereas Qwen3-VL mixes them early, partially re-segregates them, and recombines them near the output. An automated multimodal labeling pipeline assigns concise semantic descriptions to these neighborhoods. Interpolating activations toward neighborhood centroids causally redirects generation within and across modalities and outperforms difference-in-means and VL-SAE in most evaluated conditions; in one LLaVA vision-to-vision setting, MFA achieves 5.7 times the VL-SAE score. Human evaluation finds MFA steering competitive with prompting and substantially stronger than the other intervention baselines. Finally, the MFA coefficient space improves Qwen3-VL image-to-rendered-text retrieval at the deepest evaluated layer from 14.9% to 48.6% R@1. Ablations show that the reported fusion trajectories are stable across component counts, local ranks, and modality-purity thresholds. These results support local geometric neighborhoods as useful interpretable and causal units for analyzing cross-modal representations in the evaluated VLMs.

Problem

• vision-language models
• local geometry
• cross-modal representations
• interpretability
• multimodal fusion

Innovation

• local geometric decomposition
• Mixture of Factor Analyzers
• vision-language models
• cross-modal fusion
• causal interpretability