HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit

2026-03-22

AI Summary

Existing research on audio representation learning often evaluates design factors—such as input frontends, backbone architectures, and sequence lengths—in isolation, yielding conclusions that are difficult to generalize. This work proposes HELIX, a framework that systematically compares pure Mamba, pure attention, and lightweight hybrid Mamba-Attention architectures under matched parameter conditions, revealing strong coupling effects between input representations and backbone networks. HELIX effectively overcomes the memory and performance bottlenecks associated with modeling long audio sequences—up to five minutes (30,000 tokens). Evaluations across six benchmark tasks demonstrate that HELIX improves speaker identification accuracy by 11.5 percentage points over pure Mamba on five-minute utterances, while pure attention fails to run due to out-of-memory errors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Audio representation learning typically evaluates design choices such as input frontend, sequence backbone, and sequence length in isolation. We show that these axes are coupled, and conclusions from one setting often do not transfer to others. We introduce HELIX, a controlled framework comparing pure Mamba, pure attention, and a minimal hybrid with a single attention bottleneck. All models are parameter-matched at about 8.3M parameters to isolate architectural effects. Across six datasets, we find that the preferred input representation depends on the backbone, and that attention hurts performance on short, stationary audio but becomes important at longer sequence lengths. On a 5-minute speaker identification task with 30,000 tokens, pure attention fails with out-of-memory errors, while HELIX closes an 11.5-point gap over pure Mamba.

Problem

• audio representation learning
• sequence modeling
• long-context audio
• architectural coupling
• quadratic complexity

Innovation

• Mamba
• attention mechanism
• hybrid architecture
• long-sequence audio modeling
• parameter-matched comparison