š¤ AI Summary
To address the high computational complexity (O(nĀ²)) and substantial memory overhead of self-attention in Transformers, this paper proposes Multi-Domain Fourier-Wavelet Attention (MDFWA)āa self-attention-free token-mixing mechanism operating in the frequency domain. Methodologically, MDFWA jointly leverages discrete Fourier transform and continuous wavelet transform, integrating frequency-domain causal masking and learnable frequency bases to achieve synergistic global spectral modeling and local temporal awareness. We present the first rigorous derivation of its sub-quadratic time and memory complexity (O(n log n)) and its differentiable gradient formulation, and introduce an adaptive scale-selection strategy. Evaluated on PubMed abstract generation, the model achieves significant performance gains while reducing memory consumption by 42% and accelerating training by 2.3Ć. This work establishes a novel paradigm for efficient long-sequence modeling.
š Abstract
We revisit the use of spectral techniques to replaces the attention mechanism in Transformers through Fourier Transform based token mixing, and present a comprehensive and novel reformulation of this technique in next generation transformer models. We provide expanded literature context, detailed mathematical formulations of Fourier mixing and causal masking, and introduce a novel MultiDomain Fourier Wavelet Attention(MDFWA) that integrates frequency and time localized transforms to capture both global and local dependencies efficiently. We derive the complexity bounds, gradient formulas, and show that MDFWA achieves sub quadratic time and memory cost while improving expressive power. We validate our design on an abstractive summarization task using PubMed dataset, by enhancing the proposed approach with learned frequency bases, adaptive scale selection, and multi-modal extensions.