Paying Attention to Hybrid Attention: Untangling the Issues with Conversion Methods

2025-10-07

AI Summary

Existing hybrid attention methods suffer from component imbalance when converting pretrained Transformers into linear models: the linear attention branch is effectively bypassed during inference, causing the model to rely almost entirely on sliding-window softmax (SWA) and thus failing to achieve true linearization. This work is the first to identify and characterize this attribution bias. We propose a triple-coordinated strategy: (1) dynamic inference-time mixing of attention branches, (2) HedgeCATs-based weight transfer combined with LoRA fine-tuning, and (3) a stochastic softmax suppression mechanism (SSD) applied during training to actively enforce balanced participation between linear and window-based components. Component-level diagnostic analysis confirms that our approach preserves O(n) computational complexity while substantially increasing the effective contribution of linear attention—restoring over 90% of the base model’s performance. The result is an interpretable, reproducible paradigm for efficient and equitable Transformer linearization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Transformers' quadratic computational complexity limits their scalability despite remarkable performance. While linear attention reduces this to linear complexity, pre-training such models from scratch remains, in most cases, prohibitively expensive. Recent post-training linearisation methods convert pre-trained Transformers to linear models efficiently, often using hybrid approaches that combine linear attention with sliding-window softmax. We identify a critical flaw: existing hybrid methods inadvertently bypass the linear component, relying almost entirely on SWA. Component-level diagnostics reveal this previously undetected behaviour stems from overlooked evaluation practices on common-sense benchmarks. We propose three solutions to ensure balanced component usage: (i) inference-time hybridisation of linear-only conversions with sliding-window softmax; (ii) HedgeCATs, combining attention-weight transfer with targeted LoRA fine-tuning; and (iii) Scheduled Sliding-window Dropout (SSD), which stochastically suppresses the softmax branch during training to prevent component collapse. Our methods maintain computational efficiency while recovering most base model performance and ensuring genuine linear attention adoption, restoring the validity of performance attributions in hybrid conversions.

Problem

• Transformers' quadratic complexity limits model scalability
• Hybrid linear attention methods bypass the linear component
• Proposed solutions ensure balanced component usage in conversions

Innovation

• Hybridizes linear conversions with sliding-window softmax
• Combines attention-weight transfer with targeted LoRA fine-tuning
• Uses scheduled sliding-window dropout during training