When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability

2026-05-14

AI Summary

Existing methods struggle to accurately determine whether neural network modules implement identical computational mechanisms, either relying on empirical behavior while neglecting out-of-distribution mechanisms or being constrained by parameter bases that fail to account for weight-space symmetries. This work proposes a weight-based tensor similarity metric that, for the first time, formulates mechanism equivalence verification as an algebraically tractable problem. An efficient recursive algorithm enables symmetry-invariant comparison of mechanisms across layers, simultaneously preserving global functional equivalence and invariance under symmetry transformations. The method substantially outperforms existing metrics on grokking and backdoor implantation tasks, offering more precise tracking of training dynamics and significantly improving the fidelity of mechanistic similarity assessment.

Abstract

Mechanistic interpretability aims to break models into meaningful parts; verifying that two such parts implement the same computation is a prerequisite. Existing similarity measures evaluate either empirical behaviour, leaving them blind to out-of-distribution mechanisms, or basis-dependent parameters, meaning they disregard weight-space symmetries. To address these issues for the class of tensor-based models, we introduce a weight-based metric, tensor similarity, that is invariant to such symmetries. This metric captures global functional equivalence and accounts for cross-layer mechanisms using an efficient recursive algorithm. Empirically, tensor similarity tracks functional training dynamics, such as grokking and backdoor insertion, with higher fidelity than existing metrics. This reduces measuring similarity and verifying faithfulness into a solved algebraic problem rather than one of empirical approximation.

Problem

• mechanistic interpretability
• network similarity
• weight-space symmetries
• functional equivalence
• tensor-based models

Innovation

• tensor similarity
• mechanistic interpretability
• weight-space symmetries
• functional equivalence
• recursive algorithm