Simplified derivations for high-dimensional convex learning problems

2024-12-02

arXiv.org

AI Summary

This work addresses the cumbersome statistical-physics derivations and opaque physical interpretations in high-dimensional convex learning. We propose a non-replica, concise, and unified analytical framework based on the cavity method. The framework uncovers a common bipartite variable-interaction structure shared by perceptron classification—both pointwise and manifold-based—and kernel ridge regression. Leveraging a key symmetry from the perceptron capacity problem, it yields exact solutions via elementary derivations. To our knowledge, this is the first unified analysis for multi-class high-dimensional learning problems that avoids replica methods while ensuring both mathematical rigor and physical interpretability. Integrating tools from statistical physics modeling, high-dimensional random geometry, and convex optimization theory, our approach substantially simplifies derivations, reproduces, and rigorously verifies classical results on generalization error and storage capacity. It establishes a new paradigm for scalable theoretical analysis of high-dimensional learning.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Statistical-physics calculations in machine learning and theoretical neuroscience often involve lengthy derivations that obscure physical interpretation. We present concise, non-replica derivations of key results and highlight their underlying similarities. Using a cavity approach, we analyze high-dimensional learning problems: perceptron classification of points and manifolds, and kernel ridge regression. These problems share a common structure--a bipartite system of interacting feature and datum variables--enabling a unified analysis. For perceptron-capacity problems, we identify a symmetry that allows derivation of correct capacities through a na""ive method.

Problem

• Simplifies high-dimensional convex learning derivations
• Unifies analysis of perceptron and kernel regression
• Identifies symmetry in perceptron-capacity problems

Innovation

• cavity approach
• non-replica derivations
• unified bipartite system