🤖 AI Summary
Traditional Monte Carlo methods lack sufficient precision for estimating extremely small p-values in two-sample permutation tests, posing challenges for multiple hypothesis testing correction. This work proposes HAMSTest (Hash-Augmented Multilevel Splitting Test), an adaptive multilevel splitting Monte Carlo algorithm enhanced with hashing techniques. By integrating hash-based state representation with an adaptive splitting strategy, HAMSTest effectively handles the discreteness of test statistic distributions and enables efficient, high-precision estimation of arbitrarily small p-values. The method is applicable to widely used nonparametric tests such as Kolmogorov–Smirnov and Mann–Whitney U, and also supports user-defined test statistics. HAMSTest substantially improves estimation accuracy while maintaining computational efficiency and is made accessible through an open-source Python package, hamstest.
📝 Abstract
Nonparametric permutation tests are widely used for statistical analysis. However, exact computation of test p-values can be algorithmically challenging, particularly for custom tests with complex test statistics. In contrast, Monte Carlo sampling can be easily applied to any test statistic, but it suffers from poor relative accuracy when estimating small p-values, interfering with multiple hypothesis testing correction and leading to other issues. In this work, we present a hash-augmented adaptive multilevel splitting Monte Carlo algorithm that enables accurate estimation of arbitrarily small p-values in two-sample permutation tests. Using the Kolmogorov-Smirnov and the Mann-Whitney U tests as examples, we highlight potential pitfalls related to the discreteness of the test statistic distribution and show how to address them. By comparing with an exact algorithm, we demonstrate the accuracy of the p-value estimates provided by the proposed algorithm and the validity of the associated confidence intervals. We provide a reference implementation of the proposed algorithm in the Python package hamstest, which allows p-value estimation for a user-defined statistic.