retrieval-modulated attention

This work addresses the entanglement of routing and filtering functions in conventional attention mechanisms, which leads to structural opacity and optimization challenges. The authors propose S-D Attention, which explicitly decouples the interaction matrix into a low-rank routing component and a symmetric filtering component, thereby clearly distinguishing these two mechanisms for the first time. They further uncover that routing self-organizes into a spectral cascade phenomenon in deep networks. Leveraging this insight, they achieve stable training without layer normalization and validate their approach using linear attention variants (e.g., ELU+1) and effective rank analysis. Experiments show that linearizing the first seven layers of a 125M-parameter model incurs less than 5% perplexity degradation, while cascade-informed architectures reduce attention parameters by 47%–65% with only a 3.9%–8.4% increase in perplexity.

This work addresses the quadratic computational bottleneck of standard attention mechanisms in long-context scenarios and the limitations of existing hybrid attention approaches, which struggle to balance dynamic task requirements with hardware efficiency due to static allocation and head-level sparsity. The authors propose a context-aware, hierarchical dynamic hybrid attention framework that employs a parameter-efficient, lightweight Layer Router to adaptively select between full and sparse attention for each layer—without fine-tuning the pretrained large language model. This approach introduces the first layer-wise dynamic routing mechanism, effectively balancing information fidelity with memory access continuity while avoiding the hardware synchronization overhead induced by head-level sparsity. Experiments demonstrate significant performance gains over baselines on multiple long-context and mathematical reasoning benchmarks, achieving up to 2.8× speedup during prefill and 2.0× during decode, with training requiring only eight A800 GPUs for 12 hours.

Deep neural networks lack biologically plausible selective attention mechanisms, limiting both efficiency and accuracy in image recognition. To address this, we propose a spatial attention module grounded in information bottleneck theory. Our method explicitly optimizes mutual information: it minimizes the mutual information between the attention representation and the input to suppress redundancy, while maximizing the mutual information between the attention representation and task labels to enhance discriminability. Crucially, we introduce learnable anchors to quantize continuous attention scores—a novel design that strengthens information constraints and improves interpretability of attention maps. By integrating variational attention modeling with deep network embedding, our approach achieves significant performance gains across image classification, fine-grained recognition, and cross-domain classification tasks. The resulting attention maps exhibit high discriminability, strong background suppression, and enhanced interpretability.

In task arithmetic, multi-task weight coupling induces interference, degrading both training efficiency and generalization. Method: We propose a novel paradigm that fine-tunes only the attention modules of Transformers—revealing, for the first time, their intrinsic kernel-like behavior. Through systematic analysis, we identify that representation modules facilitate weight decoupling, whereas task-specific heads impede it, thereby establishing a modular decoupling design principle. Contribution/Results: Our method enhances decoupling and zero-shot task generalization without additional training. It significantly outperforms baselines across multiple benchmarks while avoiding the double training overhead required by Neural Tangent Kernel (NTK) linearization. Crucially, it achieves superior weight decoupling and single-task performance, offering a more efficient and effective alternative to existing linearized or fully fine-tuned approaches.

This work addresses the issue of diffuse attention distributions in self-attention mechanisms, which often undermine model interpretability. By analyzing the parametric structures of the query–key and output–value circuits and their differing learning rates, the study reveals that the relative learning rate critically governs attention sharpness. Through gradient flow analysis and closed-form dynamical derivations, the authors theoretically demonstrate this relationship and empirically validate it using a single-layer self-attention model. The experiments show that assigning a higher learning rate to the query–key circuit naturally induces sharper, more concentrated attention patterns. This approach significantly improves interpretability metrics while maintaining comparable predictive performance, offering a novel pathway to enhance attention focus without requiring additional regularization.

This work addresses the inefficiency of traditional token-level context modeling in distinguishing between recollective, summarizing, and local information. The authors propose a novelty-driven memory mechanism that dynamically partitions context into three components: a content-addressable novelty cache for retrievable details, a recurrent state for compressed summaries, and a sliding window for recent local context. This architecture uniquely scales memory capacity with the amount of distinct information rather than raw token count, yielding an auditable and interpretable working memory structure. Integrating a Dirichlet-process-inspired novelty-gated attention mechanism, the system achieves full-attention performance in character-level control tasks with roughly half the attention cost and outperforms both full-attention and fixed-budget baselines on a thousand-event healthcare claims prediction task, while enabling human inspection of stored memory contents.

Traditional attention mechanisms struggle to scale to long-context scenarios due to their quadratic computational complexity, while existing sparse attention methods often compromise long-range dependency modeling because of structural constraints. This work proposes MATCH, a novel framework that seamlessly integrates dynamic context retrieval into sparse attention for the first time. By leveraging efficient vector search to retrieve critical distant information in real time and dynamically fusing it with sparse attention, MATCH significantly enhances precise recall capabilities in long-context tasks without sacrificing computational efficiency. Experimental results demonstrate that MATCH consistently outperforms current sparse attention models on both synthetic and real-world natural language benchmarks, confirming its effectiveness and generality in strengthening long-range modeling capacity.

Existing approaches to long-context large language model inference often rely on fixed sparsity patterns or uniform computational budgets, overlooking the dynamic disparities among attention heads and across context positions. This work proposes EntropyInfer, a training-free framework that adaptively partitions attention heads into rigid and dynamic categories during the prefill phase based on attention entropy, enabling context-aware allocation of computational resources. During decoding, it introduces an untrained KV cache compression mechanism that preserves critical cached information aligned with the generated content. EntropyInfer achieves fine-grained, context-adaptive inference acceleration without requiring model retraining. Evaluated on Llama, Qwen, and openPangu models, it delivers up to 2.39× end-to-end speedup on sequences exceeding 100k tokens while incurring minimal quality degradation, substantially outperforming baselines such as SnapKV and AdaKV.

This study presents the first causal analysis of the internal attention mechanisms of the tabular foundation model TabPFN 2.5, investigating how it dynamically allocates computational roles across tasks of varying complexity. Employing activation patching, attention head ablation, attention entropy analysis, and contrastive activation steering on synthetic regression datasets, the work reveals that dominant attention heads exhibit 2–5 times greater causal necessity than others within specific “peak layers,” with these critical computation layers shifting in response to task complexity. In contrast, remaining attention heads display a symmetric pattern of late-stage activation. Furthermore, the research demonstrates that activation steering exhibits limited generalizability across samples, underscoring TabPFN’s strong reliance on in-context learning.