Function Spaces Without Kernels: Learning Compact Hilbert Space Representations

2025-09-24

AI Summary

This work addresses the scalability limitations and explicit kernel dependency of traditional kernel methods by proposing a novel paradigm for learning compact, adaptive Hilbert-space representations: an implicit function encoder that learns orthogonal basis functions via neural networks—eliminating the need for predefined kernels. Methodologically, we introduce two training strategies—progressive basis growth and pre-training followed by pruning—and integrate PCA to uncover intrinsic dimensionality. Theoretically, we establish equivalence between neural bases and kernel methods, derive generalization guarantees via Rademacher complexity and PAC-Bayes bounds, and define an interpretable inner-product kernel based on the learned mapping. Experiments on polynomial regression and nonlinear dynamical systems—including the Van der Pol oscillator and two-body orbital dynamics—demonstrate that our approach achieves comparable accuracy with significantly fewer basis functions, offering superior efficiency, interpretability, and kernel-level generalization performance.

Technology Category

• Machine Learning: Kernel Methods
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Function encoders are a recent technique that learn neural network basis functions to form compact, adaptive representations of Hilbert spaces of functions. We show that function encoders provide a principled connection to feature learning and kernel methods by defining a kernel through an inner product of the learned feature map. This kernel-theoretic perspective explains their ability to scale independently of dataset size while adapting to the intrinsic structure of data, and it enables kernel-style analysis of neural models. Building on this foundation, we develop two training algorithms that learn compact bases: a progressive training approach that constructively grows bases, and a train-then-prune approach that offers a computationally efficient alternative after training. Both approaches use principles from PCA to reveal the intrinsic dimension of the learned space. In parallel, we derive finite-sample generalization bounds using Rademacher complexity and PAC-Bayes techniques, providing inference time guarantees. We validate our approach on a polynomial benchmark with a known intrinsic dimension, and on nonlinear dynamical systems including a Van der Pol oscillator and a two-body orbital model, demonstrating that the same accuracy can be achieved with substantially fewer basis functions. This work suggests a path toward neural predictors with kernel-level guarantees, enabling adaptable models that are both efficient and principled at scale.

Problem

• Learning compact Hilbert space representations without kernel limitations
• Developing scalable neural basis functions independent of dataset size
• Providing kernel-style guarantees for efficient and adaptable neural models

Innovation

• Learning neural network basis functions
• Defining kernel through learned feature map
• Developing progressive training and pruning algorithms