Binary Hypothesis Testing for Softmax Models and Leverage Score Models

2024-05-09

arXiv.org

AI Summary

This paper investigates binary hypothesis testing between the Softmax and leverage-score models: given two candidate models, how to identify the true one with minimal queries. We establish the first unified statistical identifiability framework for both models and prove that their optimal sample complexity is Θ(ε⁻²). Our analysis combines information-theoretic lower bounds, classical statistical hypothesis testing, and geometric characterizations of leverage scores to rigorously establish asymptotic optimality. Crucially, we uncover a deep conceptual correspondence between the two models rooted in their probabilistic modeling principles. This work unifies hypothesis testing theory for discrete choice models and randomized linear-algebraic models, thereby providing foundational theoretical support for verifiability analysis of attention mechanisms in large language models and robustness assessment of graph algorithms.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Softmax distributions are widely used in machine learning, including Large Language Models (LLMs), where the attention unit uses softmax distributions. We abstract the attention unit as the softmax model, where given a vector input, the model produces an output drawn from the softmax distribution (which depends on the vector input). We consider the fundamental problem of binary hypothesis testing in the setting of softmax models. That is, given an unknown softmax model, which is known to be one of the two given softmax models, how many queries are needed to determine which one is the truth? We show that the sample complexity is asymptotically $O(epsilon^{-2})$ where $epsilon$ is a certain distance between the parameters of the models. Furthermore, we draw an analogy between the softmax model and the leverage score model, an important tool for algorithm design in linear algebra and graph theory. The leverage score model, on a high level, is a model which, given a vector input, produces an output drawn from a distribution dependent on the input. We obtain similar results for the binary hypothesis testing problem for leverage score models.

Problem

• Binary hypothesis testing for softmax models in machine learning
• Determine query complexity for distinguishing two softmax models
• Extend analysis to leverage score models in linear algebra

Innovation

• Binary hypothesis testing for softmax models
• Sample complexity analysis with O(ε⁻²)
• Analogous testing for leverage score models