Nonlocal Neural Tangent Kernels via Parameter-Space Interactions

2025-09-15

AI Summary

Conventional Neural Tangent Kernels (NTKs) require model parameters to be differentiable, limiting their applicability to nonsmooth objective functions, stochastic estimators, and nondifferentiable models. Method: We propose the Nonlocal Neural Tangent Kernel (NL-NTK), which replaces the classical gradient with a nonlocal gradient operator—thereby eliminating reliance on parameter differentiability. We construct two variants: a fixed-form NL-NTK and an attention-based dynamic NL-NTK, both grounded in nonlocal interaction operators that capture global dependencies in parameter space. Contribution/Results: Theoretical analysis and numerical experiments demonstrate that NL-NTK consistently characterizes training dynamics under gradient flow for both nonsmooth and stochastic models. It significantly extends the applicability of NTK theory beyond standard differentiable settings, offering enhanced theoretical generality and empirical robustness across diverse nonstandard learning scenarios.

Technology Category

• Machine Learning: Kernel Methods
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

The Neural Tangent Kernel (NTK) framework has provided deep insights into the training dynamics of neural networks under gradient flow. However, it relies on the assumption that the network is differentiable with respect to its parameters, an assumption that breaks down when considering non-smooth target functions or parameterized models exhibiting non-differentiable behavior. In this work, we propose a Nonlocal Neural Tangent Kernel (NNTK) that replaces the local gradient with a nonlocal interaction-based approximation in parameter space. Nonlocal gradients are known to exist for a wider class of functions than the standard gradient. This allows NTK theory to be extended to nonsmooth functions, stochastic estimators, and broader families of models. We explore both fixed-kernel and attention-based formulations of this nonlocal operator. We illustrate the new formulation with numerical studies.

Problem

• Extends Neural Tangent Kernel to nonsmooth functions
• Introduces nonlocal interaction-based gradient approximation
• Enables NTK theory for broader model families

Innovation

• Nonlocal interaction-based approximation in parameter space
• Extends NTK theory to nonsmooth functions
• Fixed-kernel and attention-based nonlocal operator formulations