Low-Loss Space in Neural Networks is Continuous and Fully Connected

2025-05-05

AI Summary

This work investigates the topological structure of low-loss regions in the parameter space of neural networks, specifically addressing whether distinct local minima are connected by continuous paths that maintain low loss throughout. Method: We systematically verify, for the first time in full parameter space, the connectivity between arbitrary pairs of low-loss points; we propose a novel algorithm for searching such low-loss paths. Results: Empirical evaluation on LeNet-5, ResNet-18, and Compact Convolutional Transformer demonstrates that low-loss regions form high-dimensional, continuously connected manifolds—not isolated minima. This refutes the conventional “isolated minima” hypothesis, providing new theoretical foundations for the optimization robustness and generalization of overparameterized networks. Moreover, we identify a low-loss subspace near the origin that enhances generalization. These findings unify and refine our understanding of loss landscape geometry, with implications for optimization dynamics, model interpolation, and implicit regularization.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Visualizations of the loss landscape in neural networks suggest that minima are isolated points. However, both theoretical and empirical studies indicate that it is possible to connect two different minima with a path consisting of intermediate points that also have low loss. In this study, we propose a new algorithm which investigates low-loss paths in the full parameter space, not only between two minima. Our experiments on LeNet5, ResNet18, and Compact Convolutional Transformer architectures consistently demonstrate the existence of such continuous paths in the parameter space. These results suggest that the low-loss region is a fully connected and continuous space in the parameter space. Our findings provide theoretical insight into neural network over-parameterization, highlighting that parameters collectively define a high-dimensional low-loss space, implying parameter redundancy exists only within individual models and not throughout the entire low-loss space. Additionally, our work also provides new visualization methods and opportunities to improve model generalization by exploring the low-loss space that is closer to the origin.

Problem

• Exploring continuous low-loss paths in neural network parameter space
• Demonstrating fully connected low-loss regions across architectures
• Investigating parameter redundancy and generalization in low-loss spaces

Innovation

• Explores low-loss paths in full parameter space
• Demonstrates continuous low-loss paths across architectures
• Introduces new visualization methods for low-loss space