Interior interpretability with attention rollout: contraction and propagation profiles in Transformers

2026-07-24

AI Summary

Existing feature attribution methods struggle to characterize how attention mechanisms in intermediate Transformer layers propagate and combine feature information. This work proposes an “internal interpretability” framework that, for the first time, introduces Doeblin–Dobrushin contraction theory into attention analysis by modeling attention rollout as a row-stochastic operator and establishing its quantitative relationship with rank-one matrices, thereby endowing the propagation process with a rigorous mathematical interpretation. Experiments on a metabolomic age-prediction Transformer reveal that rollout contractivity intensifies with network depth, and trained models exhibit markedly different propagation profiles compared to randomly initialized ones. Although the method shows local agreement with PCA and GradientExplainer on top-ranked features, overall ranking correlations remain weak.

Abstract

Feature-attribution methods assign scores relating input variables to a model's output, but do not by themselves characterize how explicitly defined interaction operators compose across its intermediate layers. We introduce \emph{interior interpretability}, a propagation-based perspective on internal model organization, and instantiate it for tabular Transformers using attention rollout. We interpret rollout as a row-stochastic operator encoding attention-mediated propagation between feature tokens. By applying classical Doeblin--Dobrushin contraction theory, we show that a rollout operator with a small Dobrushin coefficient is quantitatively close to a rank-one stochastic matrix whose common row is determined by its normalized column sums. This result gives a structural interpretation to the corresponding rollout propagation profile. In Transformers trained for metabolomic age prediction, the measured rollout contraction strengthens with depth. Trained and randomly initialized models also exhibit different propagation profiles, although the present experiments do not establish the predictive relevance of individual rollout-ranked variables. Exploratory comparisons with PCA and GradientExplainer approximations to SHAP reveal localized agreement among highly ranked variables but weak agreement across complete rankings. Attention rollout is therefore used here as a diagnostic of attention-mediated propagation, not as a causal explanation or faithful attribution of the complete Transformer.

Problem

• interior interpretability
• attention rollout
• feature attribution
• Transformer
• propagation profile

Innovation

• interior interpretability
• attention rollout
• Dobrushin contraction
• propagation profile
• row-stochastic operator