Prescriptive SVD-Inspired Attention via Spectral Energy Retention

2026-09-21

AI Summary

本文通过在注意力机制中引入谱能量保留方法，解决了自注意力机制中难以识别重要结构方向的问题，实现了模型的可解释性和参数优化。

Abstract

Self-attention is central to modern Transformer architectures, but its dense dot-product formulation makes it difficult to identify which internal directions are structurally important and which can be modified without disrupting the model. SVD-Inspired Attention (SVDA) addresses part of this problem by introducing a learned diagonal spectrum into the query-key score interaction, making latent attention directions explicitly inspectable through indicators such as spectral entropy, effective rank, sparsity, alignment, selectivity, and perturbation response. This paper examines the transition from diagnostic interpretation to operational intervention. A diagnosis--intervention--verification framework is proposed, and one intervention is evaluated: spectral energy retention in the attention-score pathway. Across FashionMNIST, CIFAR-10, CIFAR-100, and Food-101, the $ρ=0.90$ prescription removes 24.5--53.7\% of score directions, reduces parameters by 2.6--4.3\%, and reduces estimated MACs by 2.8--5.4\%. The paired mean accuracy change of the dimension-reduced model ranges from $-0.03$ to $+0.05$ percentage points over three seeds. These results support SVDA as an intrinsically interpretable attention mechanism whose learned spectrum exposes an operational coordinate system for deterministic and verifiable modification of attention-score formation.

Problem

• self-attention
• SVD-Inspired Attention
• spectral energy retention
• diagnostic interpretation
• operational intervention

Innovation

• SVD-Inspired Attention
• spectral energy retention
• diagnosis-intervention-verification framework