Tensor Network Based Feature Learning Model

2025-12-02

International Conference on Artificial Intelligence and Statistics

AI Summary

To address the high computational cost of hyperparameter tuning via cross-validation in kernel methods, this paper proposes a tensor network-based learnable feature modeling framework. The core innovation lies in embedding hyperparameter learning directly into the tensor structure: a learnable CP decomposition explicitly models tensor-product features—such as polynomial and Fourier bases—enabling joint optimization of hyperparameters and model parameters; training is efficiently performed via alternating least squares. This work presents the first end-to-end automatic learning of kernel feature hyperparameters within tensor networks, eliminating the repeated model retraining inherent in conventional cross-validation. Experiments demonstrate that the method achieves comparable prediction accuracy while accelerating training by 3–5×, significantly enhancing both efficiency and scalability of feature learning in large-scale kernel methods.

Technology Category

• Machine Learning: Kernel Methods
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Many approximations were suggested to circumvent the cubic complexity of kernel-based algorithms, allowing their application to large-scale datasets. One strategy is to consider the primal formulation of the learning problem by mapping the data to a higher-dimensional space using tensor-product structured polynomial and Fourier features. The curse of dimensionality due to these tensor-product features was effectively solved by a tensor network reparameterization of the model parameters. However, another important aspect of model training - identifying optimal feature hyperparameters - has not been addressed and is typically handled using the standard cross-validation approach. In this paper, we introduce the Feature Learning (FL) model, which addresses this issue by representing tensor-product features as a learnable Canonical Polyadic Decomposition (CPD). By leveraging this CPD structure, we efficiently learn the hyperparameters associated with different features alongside the model parameters using an Alternating Least Squares (ALS) optimization method. We prove the effectiveness of the FL model through experiments on real data of various dimensionality and scale. The results show that the FL model can be consistently trained 3-5 times faster than and have the prediction quality on par with a standard cross-validated model.

Problem

• Learning optimal feature hyperparameters efficiently
• Overcoming curse of dimensionality in tensor-product features
• Accelerating training while maintaining prediction quality

Innovation

• Tensor network reparameterization solves dimensionality curse
• CPD representation enables learnable hyperparameters for features
• ALS optimization jointly trains model and feature parameters