Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces

2025-01-30

AI Summary

This work addresses the challenge of rapidly generalizing models to new tasks in transfer learning. We propose a geometric transfer learning framework grounded in Hilbert space theory. Methodologically, we introduce the first functional-space geometric characterization of transfer learning, unifying interpolation, linear extrapolation, and nonlinear extrapolation under a single theoretical umbrella. We further propose a novel function encoder paradigm, rigorously proving its universal approximation property, and integrate kernel methods with functional analysis to enable efficient least-squares training. Empirically, our approach achieves state-of-the-art performance across four benchmark tasks, consistently outperforming leading methods—including Transformers and meta-learning approaches—both in generalization accuracy and adaptation speed. Key contributions include: (i) a principled geometric formalism for transfer learning in reproducing kernel Hilbert spaces; (ii) a theoretically grounded, trainable function encoder; and (iii) empirical validation demonstrating superior sample efficiency and cross-task generalization.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

A central challenge in transfer learning is designing algorithms that can quickly adapt and generalize to new tasks without retraining. Yet, the conditions of when and how algorithms can effectively transfer to new tasks is poorly characterized. We introduce a geometric characterization of transfer in Hilbert spaces and define three types of inductive transfer: interpolation within the convex hull, extrapolation to the linear span, and extrapolation outside the span. We propose a method grounded in the theory of function encoders to achieve all three types of transfer. Specifically, we introduce a novel training scheme for function encoders using least-squares optimization, prove a universal approximation theorem for function encoders, and provide a comprehensive comparison with existing approaches such as transformers and meta-learning on four diverse benchmarks. Our experiments demonstrate that the function encoder outperforms state-of-the-art methods on four benchmark tasks and on all three types of transfer.

Problem

• Adaptive Learning
• Computer Programs
• Task Flexibility

Innovation

• Function Encoder
• Adaptive Learning
• Universal Function Approximation