Traversing the solution space of neural networks with Hessian Null Space Continuation

2026-09-29

AI Summary

This study addresses the unclear computational diversity within locally mode-connected regions of neural networks and its relationship to weight space. To investigate this, it proposes Hessian Null-space Continuation (HNC), a method that leverages local curvature to traverse weight regions preserving network functionality, thereby unifying the study of mode connectivity and representational collapse for the first time. The work reveals substantial representational diversity and markedly distinct internal mechanisms within the neighborhood of a single solution. This high diversity is validated across recurrent neural networks, vision transformers, and reinforcement learning settings. Furthermore, by quantifying the geometric structure of solution sets, the findings offer novel perspectives for model editing.

Abstract

On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network than any independently trained model with a different architecture or objective. In reinforcement-learning agents, HNC uncovers a distinct navigation strategy at comparable return and exposes reward hacking in an AI Safety Gridworld. Finally, HNC measures the local geometry of the solution set, showing how model size and task complexity shape its dimension and functional sensitivity. Our results show that a surprisingly large amount of representational diversity exists near a single trained solution, unseen by standard gradient-based optimization. HNC identifies and quantifies this diversity, opening new possibilities for mechanistic understanding of solution spaces and for model merging, editing, and fine-tuning.

Problem

• solution space
• mode connectivity
• representation degeneracy
• neural network internal mechanisms
• weight space

Innovation

• Hessian Null Space Continuation
• solution space traversal
• representational diversity
• mode connectivity
• mechanistic interpretability