Trading off Consistency and Dimensionality of Convex Surrogates for the Mode

2024-02-16

arXiv.org

AI Summary

To address the intractability of surrogate loss optimization in large-scale multiclass classification caused by high-dimensional embeddings, this paper proposes a low-dimensional convex polyhedral embedding framework, establishing theoretical trade-offs among embedding dimension, consistency regions, and data distribution assumptions. First, it rigorously proves that “hallucination”—i.e., spurious class predictions—necessarily occurs when the embedding dimension is less than $n-1$. Second, under a low-noise assumption, it derives a verifiable consistency criterion. Third, it designs structured embeddings—including hypercubes and permutahedra—that achieve dimensionality reductions from $2^d$ to $d$ and from $d!$ to $d$, respectively. Finally, in the multiple-instance learning setting, it shows that full simplex consistency is guaranteed with only $n/2$ embedding dimensions and proves the existence of consistent subsets around any point-mass distribution.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In multiclass classification over $n$ outcomes, the outcomes must be embedded into the reals with dimension at least $n-1$ in order to design a consistent surrogate loss that leads to the"correct"classification, regardless of the data distribution. For large $n$, such as in information retrieval and structured prediction tasks, optimizing a surrogate in $n-1$ dimensions is often intractable. We investigate ways to trade off surrogate loss dimension, the number of problem instances, and restricting the region of consistency in the simplex for multiclass classification. Following past work, we examine an intuitive embedding procedure that maps outcomes into the vertices of convex polytopes in a low-dimensional surrogate space. We show that full-dimensional subsets of the simplex exist around each point mass distribution for which consistency holds, but also, with less than $n-1$ dimensions, there exist distributions for which a phenomenon called hallucination occurs, which is when the optimal report under the surrogate loss is an outcome with zero probability. Looking towards application, we derive a result to check if consistency holds under a given polytope embedding and low-noise assumption, providing insight into when to use a particular embedding. We provide examples of embedding $n = 2^{d}$ outcomes into the $d$-dimensional unit cube and $n = d!$ outcomes into the $d$-dimensional permutahedron under low-noise assumptions. Finally, we demonstrate that with multiple problem instances, we can learn the mode with $frac{n}{2}$ dimensions over the whole simplex.

Problem

• Reducing surrogate loss dimension for large multiclass classification problems
• Avoiding hallucination when using low-dimensional convex polytope embeddings
• Trading off consistency, dimensionality, and number of problem instances

Innovation

• Low-dimensional convex polytope embeddings for surrogates
• Consistency analysis under low-noise assumptions
• Multiple instances enable mode learning with reduced dimensions