🤖 AI Summary
This work addresses the inefficiency and performance degradation caused by standard LoRA’s uniform adaptation across heterogeneous components in mixture-of-experts language models, which overlooks the functional distinctions between attention and recurrent modules. The authors propose a component-aware LoRA adaptation strategy that differentially deploys adapters in sequential (Qwen3.5-0.8B) and parallel (Falcon-H1-0.5B) hybrid architectures. Experimental results demonstrate that adapting only the attention pathways achieves superior performance over full fine-tuning with 5–10× fewer parameters. Moreover, adapter application to recurrent components proves detrimental in sequential architectures—causing a 14.8% performance drop—but beneficial in parallel ones, yielding an 8.6% gain. The parallel architecture further exhibits positive cross-task transferability, whereas the sequential variant suffers from catastrophic forgetting.
📝 Abstract
Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniformly without considering the distinct functional roles of each component type. We systematically study component-type LoRA placement across two hybrid architectures -- Qwen3.5-0.8B (sequential, GatedDeltaNet + softmax attention) and Falcon-H1-0.5B (parallel, Mamba-2 SSM + attention) -- fine-tuned on three domains and evaluated on five benchmarks. We find that the attention pathway -- despite being the minority component -- consistently outperforms full-model adaptation with 5-10x fewer trainable parameters. Crucially, adapting the recurrent backbone is destructive in sequential hybrids (-14.8 pp on GSM8K) but constructive in parallel ones (+8.6 pp). We further document a transfer asymmetry: parallel hybrids exhibit positive cross-task transfer while sequential hybrids suffer catastrophic forgetting. These results establish that hybrid topology fundamentally determines adaptation response, and that component-aware LoRA placement is a necessary design dimension for hybrid architectures.