Second-Order Asymptotics of Two-Sample Tests

2026-01-14

AI Summary

This work addresses the problem of minimizing the type-II error exponent in two-sample testing while controlling the type-I error probability. It proposes a generalized “divergence test” framework that extends Gutman’s original test—based on Jensen–Shannon divergence—to arbitrary information divergences. Within this framework, all divergence tests are shown to achieve the optimal first-order error exponent; moreover, when invariant divergences are employed, the second-order asymptotic performance of Gutman’s test is recovered. The study further reveals that Gutman’s test is in fact the generalized likelihood ratio test for this setting and establishes a theoretical connection between two-sample testing and robust goodness-of-fit testing, thereby unifying and extending classical results in the field.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In two-sampling testing, one observes two independent sequences of independent and identically distributed random variables distributed according to the distributions $P_1$ and $P_2$ and wishes to decide whether $P_1=P_2$ (null hypothesis) or $P_1\neq P_2$ (alternative hypothesis). The Gutman test for this problem compares the empirical distributions of the observed sequences and decides on the null hypothesis if the Jensen-Shannon (JS) divergence between these empirical distributions is below a given threshold. This paper proposes a generalization of the Gutman test, termed \emph{divergence test}, which replaces the JS divergence by an arbitrary divergence. For this test, the exponential decay of the type-II error probability for a fixed type-I error probability is studied. First, it is shown that the divergence test achieves the optimal first-order exponent, irrespective of the choice of divergence. Second, it is demonstrated that the divergence test with an invariant divergence achieves the same second-order asymptotics as the Gutman test. In addition, it is shown that the Gutman test is the GLRT for the two-sample testing problem, and a connection between two-sample testing and robust goodness-of-fit testing is established.

Problem

• two-sample testing
• hypothesis testing
• Jensen-Shannon divergence
• type-II error
• goodness-of-fit testing

Innovation

• divergence test
• second-order asymptotics
• two-sample testing
• GLRT
• invariant divergence