🤖 AI Summary
This work addresses the unclear efficacy of attention mechanisms versus state space models (SSMs) or linear attention components in current hybrid language architectures. To systematically evaluate the necessity and functional division of these modules, the authors propose a functional component ablation framework employing group-wise ablation, layer-wise scanning, positional ablation, and randomized controlled trials across multiple benchmarks. Their analysis provides the first quantitative evidence that SSMs or linear attention serve as the true modeling backbone: their removal degrades perplexity by over 35,000-fold—far exceeding the 82-fold degradation from removing attention. The study further uncovers positional gradient effects, functional redundancy across components, and demonstrates that hybrid models exhibit 20–119 times greater robustness to random layer removal than pure Transformers.
📝 Abstract
Hybrid language models combining attention with state space models (SSMs) or linear attention offer improved efficiency, but whether both components are genuinely utilized remains unclear. We present a functional component ablation framework applied to two sub-1B hybrid models -- Qwen3.5-0.8B (sequential: Gated DeltaNet + softmax attention) and Falcon-H1-0.5B (parallel: Mamba-2 + attention) -- with a pure Transformer control (Qwen2.5-0.5B). Through group ablations, layer-wise sweeps, positional ablations, matched random controls, and perplexity analysis across five benchmarks, we establish four findings: (1) both component types are essential and neither is bypassed; (2) the alternative component (linear attention or SSM) is the primary language modeling backbone, causing >35,000x perplexity degradation when removed versus ~82x for attention; (3) component importance follows a positional gradient, with early layers being disproportionately critical; and (4) hybrid architectures exhibit 20-119x greater resilience to random layer removal than pure Transformers, revealing built-in functional redundancy between component types. These results provide actionable guidance for hybrid model compression, architecture design, and fault-tolerant deployment.