Exact statistical tests using integer programming: Leveraging an overlooked approach for maximizing power for differences between binomial proportions

2025-03-17

AI Summary

In small-sample two-sample binomial proportion testing, conventional methods suffer from a power imbalance: the Wald test is overly liberal (inflated Type I error), while Fisher’s exact test is overly conservative (reduced power). Method: This paper proposes a novel exact two-sample testing framework based on integer programming. Extending a classic 1969 approach, it operates directly on the discrete null sample space to strictly control Type I error, and constructs the rejection region via combinatorial optimization to maximize weighted average power—supporting customizable prior weights. Contribution/Results: We prove that the method guarantees exact Type I error control and achieves average power that is optimal or near-optimal under small samples. Empirical evaluations demonstrate substantial improvements over standard benchmarks—including Wald and Fisher’s exact tests—yielding superior balance between statistical power and robustness.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Traditional hypothesis testing methods for differences in binomial proportions can either be too liberal (Wald test) or overly conservative (Fisher's exact test), especially in small samples. Regulators favour conservative approaches for robust type I error control, though excessive conservatism may significantly reduce statistical power. We offer fundamental theoretical contributions that extend an approach proposed in 1969, resulting in the derivation of a family of exact tests designed to maximize a specific type of power. We establish theoretical guarantees for controlling type I error despite the discretization of the null parameter space. This theoretical advancement is supported by a comprehensive series of experiments to empirically quantify the power advantages compared to traditional hypothesis tests. The approach determines the rejection region through a binary decision for each outcome dataset and uses integer programming to find an optimal decision boundary that maximizes power subject to type I error constraints. Our analysis provides new theoretical properties and insights into this approach's comparative advantages. When optimized for average power over all possible parameter configurations under the alternative, the method exhibits remarkable robustness, performing optimally or near-optimally across specific alternatives while maintaining exact type I error control. The method can be further customized for particular prior beliefs by using a weighted average. The findings highlight both the method's practical utility and how techniques from combinatorial optimization can enhance statistical methodology.

Problem

• Develop exact two-sample binomial tests with maximum power
• Address liberal or conservative issues in traditional binomial tests
• Ensure type I error control while optimizing power

Innovation

• Integer programming optimizes exact binomial tests
• Maximizes power under type I error constraints
• Guarantees robust error control via combinatorial optimization