Comparing three learn-then-test paradigms in a multivariate normal means problem

2026-01-12

AI Summary

This study addresses the trade-off between selection bias and statistical power in the “learn-then-test” paradigm for multivariate normal mean testing. It systematically compares three prominent correction strategies: sample splitting, null augmentation, and resampling-based calibration. The work establishes, for the first time, a unified asymptotic power framework encompassing all three approaches. Within this framework, it reveals that null augmentation achieves near-optimal power when the number of augmented null hypotheses scales with the square root of the total number of hypotheses—a finding that challenges prevailing heuristic practices. Theoretical analysis further demonstrates that resampling-based correction yields the highest power, while null augmentation offers a close yet more computationally tractable alternative. Moreover, the optimal splitting proportion for sample splitting depends critically on the difficulty of the underlying structure learning task.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Many modern procedures use the data to learn a structure and then leverage it to test many hypotheses. If the entire data is used at both stages, analytical or computational corrections for selection bias are required to ensure validity (post-learning adjustment). Alternatively, one can learn and/or test on masked versions of the data to avoid selection bias, either via information splitting or null augmentation}. Choosing among these three learn-then-test paradigms, and how much masking to employ for the latter two, are critical decisions impacting power that currently lack theoretical guidance. In a multivariate normal means model, we derive asymptotic power formulas for prototypical methods from each paradigm -- variants of sample splitting, conformal-style null augmentation, and resampling-based post-learning adjustment -- quantifying the power losses incurred by masking at each stage. For these paradigm representatives, we find that post-learning adjustment is most powerful, followed by null augmentation, and then information splitting. Moreover, null augmentation can be nearly as powerful as post-learning adjustment, while avoiding its challenges: the power of the former approaches that of the latter if the number of nulls used for augmentation is a vanishing fraction of the number of hypotheses. We also prove for a tractable proxy that the optimal number of nulls scales as the square root of the number of hypotheses, challenging existing heuristics. Finally, we characterize optimal tuning for information splitting by identifying an optimal split fraction and tying it to the difficulty of the learning problem. These results establish a theoretical foundation for key decisions in the deployment of learn-then-test methods.

Problem

• learn-then-test
• selection bias
• multiple hypothesis testing
• data masking
• statistical power

Innovation

• learn-then-test
• selection bias correction
• null augmentation
• asymptotic power analysis
• information splitting