🤖 AI Summary
This work addresses a fundamental paradox in hybrid recurrent-attention architectures: content routing seeks to bypass the computational expense of attention, yet relies on attention-derived representations to function effectively. Through over twenty controlled experiments, the study systematically investigates the representational conditions necessary for efficient routing and reveals, for the first time, that attention not only serves as a computational mechanism but also constructs a low-dimensional, routable subspace by encoding pairwise matching outcomes into representations. Empirical results demonstrate that a single-layer softmax attention module can induce a subspace of approximately 34 dimensions, achieving 98.4% routing accuracy—dramatically higher than the 1.2% obtained without attention. This subspace cannot be replicated by random projections or contrastive learning, while classical non-learning methods such as Bloom filters attain only 90.9% accuracy, underscoring the irreplaceable role of attention in shaping effective routing representations.
📝 Abstract
We identify a routing paradox in hybrid recurrent-attention architectures: content-based routing - deciding which tokens deserve expensive attention - requires exactly the pairwise computation that routing is designed to avoid. Through 20+ controlled experiments across three tasks (a synthetic diagnostic, the Zoology MQAR benchmark, and HotpotQA), we map the routing landscape exhaustively. One layer of softmax attention creates a latent ~34-dimensional subspace enabling 98.4% routing precision; zero layers yield 1.2%. This subspace is invisible to cosine similarity, destroyed by random projections (98.4% to 2.6%), and cannot be created by contrastive pretraining - proving attention's role is writing pairwise match results into representations, not merely computing them. Twelve alternative mechanisms all cluster at 15-29%. Non-learned indices (Bloom filter: 90.9%; BM25 on HotpotQA: 82.7%) bypass the bottleneck entirely. The result is a sharp two-regime hierarchy with an empty middle ground. These findings provide the mechanistic explanation for the empirical observation that recurrent models fail at associative recall, and reframe attention as a representation constructor rather than merely a computation mechanism.