Dual Attention Residuals

2026-07-21

AI Summary

This work addresses the limited dynamic depth selection in conventional Transformer residual architectures, which stems from insufficient historical information exchange across multiple parallel streams. To overcome this, we propose a reciprocal cross-stream addressing mechanism that enables bidirectional historical retrieval within a multi-stream framework: each stream computes depth weights based on the state of its counterpart stream and applies these weights to its own historical values. Our approach uniquely integrates inter-stream interaction into historical retrieval, preserving inter-layer representational diversity through cross-stream depth selection while mitigating redundancy and functional imbalance. Key components include reciprocal cross-attention, normalized state weighting, constrained gated writing, and block-level history storage. Experiments demonstrate consistent and significant improvements over standard residual Transformers and Attention Residuals across dense models (0.1B–1B) and a 7B sparse MoE model, with ablation studies confirming that performance gains arise from cross-stream interaction rather than additional parameters or projections.

Abstract

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Problem

• historical retrieval
• multi-stream residuals
• cross-stream interaction
• depth selection
• Transformer architecture

Innovation

• Dual Attention Residuals
• multi-stream interaction
• historical retrieval
• cross-stream addressing
• depth-wise diversity