Can Kernel Methods Explain How the Data Affects Neural Collapse?

2024-06-04

AI Summary

This work investigates how data distribution influences intra-class variability reduction (NC1), a core phenomenon in neural collapse (NC), addressing the limitation of existing unconstrained feature models that neglect data-dependent effects. Method: We propose a kernel-based analytical framework—incorporating the Neural Tangent Kernel (NTK), Neural Network Gaussian Process (NNGP) kernel, and a novel data-aware Gaussian process kernel—to rigorously characterize NC1 under realistic data assumptions. Contribution/Results: We theoretically prove that, for Gaussian data, NTK does not outperform NNGP in capturing NC1, exposing inherent limitations of data-agnostic kernels. Empirically, we demonstrate that activation functions critically modulate NC1 (ERF surpasses ReLU), and we provide the first empirical validation that data-aware kernels mitigate NC1—though they do not necessarily replicate NC1 trends observed in shallow networks. Our work advances data-adaptive kernel theory for neural collapse, delivers unified empirical insights, and releases open-source code.

Technology Category

• Machine Learning: Kernel Methods
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

A vast amount of literature has recently focused on the"Neural Collapse"(NC) phenomenon, which emerges when training neural network (NN) classifiers beyond the zero training error point. The core component of NC is the decrease in the within-class variability of the network's deepest features, dubbed as NC1. The theoretical works that study NC are typically based on simplified unconstrained features models (UFMs) that mask any effect of the data on the extent of collapse. To address this limitation of UFMs, this paper explores the possibility of analyzing NC1 using kernels associated with shallow NNs. We begin by formulating an NC1 metric as a function of the kernel. Then, we specialize it to the NN Gaussian Process kernel (NNGP) and the Neural Tangent Kernel (NTK), associated with wide networks at initialization and during gradient-based training with a small learning rate, respectively. As a key result, we show that the NTK does not represent more collapsed features than the NNGP for Gaussian data of arbitrary dimensions. This showcases the limitations of data-independent kernels such as NTK in approximating the NC behavior of NNs. As an alternative to NTK, we then empirically explore a recently proposed data-aware Gaussian Process kernel, which generalizes NNGP to model feature learning. We show that this kernel yields lower NC1 than NNGP but may not follow the trends of the shallow NN. Our study demonstrates that adaptivity to data may allow kernel-based analysis of NC, though further advancements in this area are still needed. A nice byproduct of our study is showing both theoretically and empirically that the choice of nonlinear activation function affects NC1 (with ERF yielding lower values than ReLU). The code is available at: https://github.com/kvignesh1420/shallow_nc1

Problem

• Analyzing Neural Collapse (NC1) using kernel methods for shallow neural networks
• Comparing NNGP and NTK kernels in approximating NC behavior with Gaussian data
• Exploring data-aware kernels for NC analysis and impact of activation functions

Innovation

• Uses NNGP and NTK kernels for NC1 analysis
• Explores data-aware Gaussian Process kernel
• Shows activation function affects NC1 values