Spectral Superposition: A Theory of Feature Geometry

2026-02-02

AI Summary

This work addresses a critical oversight in existing approaches: the neglect of the geometric structure of the feature-sharing space within neural network hyperspherical representations. It introduces, for the first time, a spectral framework that leverages the frame operator $F = WW^\top$, derived from weight matrices, to establish a spectral measure theory characterizing norm distribution of features across subspaces. This reveals the global geometric relationships among features. By integrating operator theory, spectral analysis, frame theory, and association schemes, the study proposes spectral localization, tight-frame organization, and discrete categorization to unify diverse geometric configurations. In a toy model, it demonstrates that capacity saturation forces features to collapse into a single subspace and form a tight frame. The framework generalizes to real-world networks, offering a novel paradigm for interpretability.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Representation Learning for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Social media analysis through the lenses of networks
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Neural networks represent more features than they have dimensions via superposition, forcing features to share representational space. Current methods decompose activations into sparse linear features but discard geometric structure. We develop a theory for studying the geometric structre of features by analyzing the spectra (eigenvalues, eigenspaces, etc.) of weight derived matrices. In particular, we introduce the frame operator $F = WW^\top$, which gives us a spectral measure that describes how each feature allocates norm across eigenspaces. While previous tools could describe the pairwise interactions between features, spectral methods capture the global geometry (``how do all features interact?''). In toy models of superposition, we use this theory to prove that capacity saturation forces spectral localization: features collapse onto single eigenspaces, organize into tight frames, and admit discrete classification via association schemes, classifying all geometries from prior work (simplices, polygons, antiprisms). The spectral measure formalism applies to arbitrary weight matrices, enabling diagnosis of feature localization beyond toy settings. These results point toward a broader program: applying operator theory to interpretability.

Problem

• superposition
• feature geometry
• spectral analysis
• neural network interpretability
• frame operator

Innovation

• spectral superposition
• frame operator
• feature geometry
• tight frames
• operator theory