appearance-conditioned cross-attention

This work investigates how to explicitly control and quantify the information flow within the attention mechanism of Vision Transformers to uncover the evolutionary process from independent local patch processing to the emergence of global representations. The authors introduce a variational information bottleneck along all pathways where attention writes into the residual stream, without altering the model architecture. This approach models the transmitted information as an explicit, tunable variable, enabling precise constraint and intervention on internal communication. Experiments on ImageNet-100 demonstrate that the method effectively modulates the degree of attention coordination and reveals a clear relationship between classification performance and information routing strategies, offering a novel perspective for understanding the internal mechanisms of Vision Transformers.

This work investigates the roles of masked-patch self-attention and masked-to-visible cross-attention in the MAE decoder for representation learning, revealing that image reconstruction primarily relies on global semantic representations extracted by the encoder—not on intra-masked-patch interactions within the decoder. Motivated by this finding, we propose CrossMAE: a streamlined framework that retains only the cross-attention mechanism while entirely removing self-attention among masked tokens in the decoder. This design is the first to empirically demonstrate that MAE’s effectiveness stems from the encoder’s strong global modeling capacity, challenging the prevailing assumption that decoder-side modeling of dependencies among masked patches is essential. Evaluated across ViT-S to ViT-H architectures, CrossMAE matches or surpasses standard MAE in performance while reducing GPU memory consumption by 37% and FLOPs by 42%. Code and pretrained models are publicly available.

This work addresses the prevalent hallucination issues in current large vision-language models when handling multi-image tasks, which stem primarily from the locality of attention mechanisms and insufficient cross-image modeling capabilities. To mitigate this, the authors propose a co-optimization framework that integrates architectural and training-level innovations. At the architecture level, an optional image-token interaction attention mechanism is introduced to enable fine-grained cross-image alignment. At the training level, a cross-image contrastive preference learning strategy is designed to reinforce the model’s reliance on authentic visual evidence. This approach represents the first effort to jointly suppress multi-image hallucinations through coordinated structural and objective design, achieving significant performance gains across diverse multi-image tasks while maintaining or slightly improving single-image task performance, thereby demonstrating strong generalization capability.

This work addresses the problem of controllable multi-image co-generation, where the goal is to generate multiple images sharing a consistent background while enabling the central object to vary flexibly according to distinct text prompts. The proposed method is a text-driven diffusion model that explicitly decouples background and foreground representations. Its key contributions are: (1) the first introduction of a time-varying weight decoupling mechanism within the cross-attention layers of diffusion models, enabling explicit separation of background and foreground features; and (2) a multi-objective sampling optimization framework that jointly enhances background coupling, text–image alignment, and visual fidelity. Extensive experiments demonstrate that the method significantly outperforms existing approaches in background consistency, text fidelity, and image quality. By enabling precise, prompt-conditioned foreground manipulation without compromising background coherence, it establishes a novel paradigm for controllable image generation.

Deep neural networks lack biologically plausible selective attention mechanisms, limiting both efficiency and accuracy in image recognition. To address this, we propose a spatial attention module grounded in information bottleneck theory. Our method explicitly optimizes mutual information: it minimizes the mutual information between the attention representation and the input to suppress redundancy, while maximizing the mutual information between the attention representation and task labels to enhance discriminability. Crucially, we introduce learnable anchors to quantize continuous attention scores—a novel design that strengthens information constraints and improves interpretability of attention maps. By integrating variational attention modeling with deep network embedding, our approach achieves significant performance gains across image classification, fine-grained recognition, and cross-domain classification tasks. The resulting attention maps exhibit high discriminability, strong background suppression, and enhanced interpretability.

This work investigates how vision-language models (VLMs) attend to relevant image regions when generating captions, revealing a class of sparse attention heads—termed “gaze heads”—whose attention distributions closely align with the spatial locations of the described content. Through multi-scale analysis of VLM attention mechanisms, the study demonstrates that intervening on fewer than 9% of these gaze heads can precisely steer caption generation toward specified image regions without fine-tuning. By integrating attention relevance analysis with targeted masking, the method achieves 83.1% accuracy in region-directed captioning on both comics and COCO benchmarks and enables dynamic switching of focus during generation. This mechanism proves robust across model scales from 2B to 32B parameters and multiple VLM architectures.

研究通过提出SHS-Index量化视觉Transformer注意力头的语义专业化，设计了Ariadne Attention以减少计算同时保持性能。

This study addresses the lack of mechanistic understanding regarding object hallucination in VQ-tokenized vision-language models by attributing hallucinations to specific architectural circuits rather than generic calibration biases. Through activation patching, a three-gate diagnostic framework, and univariate architectural substitution experiments, we identify early attention routing circuits across architectures as the locus of hallucination, confirming vector quantization as its source and proposing targeted interventions. Experiments demonstrate that ablating only the L0 layer reduces CHAIRi, an open-ended generation hallucination metric, by 31%, significantly outperforming decoding-time mitigation methods such as VCD and DoLA. This work establishes a novel mechanism-driven perspective for hallucination research.

This work addresses functional mismatch and redundancy in the attention mechanisms of current large vision-language models, which fail to efficiently exploit visual context. By establishing a unified framework grounded in information theory and information geometry, the study quantifies the geometric structure and entropy characteristics of residual updates, revealing a functional decoupling between attention mechanisms and feed-forward networks (FFNs) in subspace operations. For the first time from an information-geometric perspective, it clarifies their distinct intrinsic roles and demonstrates that attention can be replaced by predefined weights—such as those derived from Gaussian noise—without performance degradation. Empirical results show that this simplified model matches or even surpasses the original architecture across multiple benchmarks, challenging the prevailing design paradigm reliant on dynamic attention and confirming its substantial redundancy.

This study addresses the limitation of existing video encoding models that overlook the spatiotemporal structure of neural responses, hindering the prediction of dynamic visual processing mechanisms in the brain under naturalistic video conditions. We propose the first video-oriented joint spatiotemporal cross-attention framework, which leverages features from the self-supervised model V-JEPA-2 for spatiotemporally joint routing to construct an interpretable fMRI encoding model simulating the cortex’s adaptive weighting of dynamic information. This model significantly improves brain response prediction accuracy in higher-order visual areas and generates interpretable attention maps capable of tracking moving objects. Furthermore, it reveals dynamically shifting attention allocation patterns within motion perception and semantic selection networks as objects move, thereby elucidating the computational mechanisms underlying dynamic visual perception.