SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation

2026-07-27

AI Summary

Standard Transformers in recommender systems suffer from spectral collapse in embeddings and attention mechanisms, limiting their performance compared to lightweight models. Addressing the spectral concentration induced by data heterogeneity and long-tailed distributions, this work proposes the first spectrum-aware Transformer architecture. It introduces learnable spectral softening modules, spectrally smoothed attention mechanisms, and a spectral residual positional encoding derived from Taylor expansion of singular values, thereby incorporating spectral uniformity as an inductive bias into recommendation modeling. By jointly leveraging spectral regularization and singular value distribution control, the proposed method significantly outperforms existing state-of-the-art approaches across multiple public and industrial datasets. Moreover, in real-world deployment, it consistently enhances both the effective rank of attention and recommendation performance as model depth increases.

Abstract

Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.

Problem

• embedding collapse
• attention collapse
• spectral collapse
• recommendation
• Transformer

Innovation

• Spectral Softening
• Attention Collapse
• Embedding Collapse
• Spectrum-aware Transformer
• Effective Rank