🤖 AI Summary
This work addresses the trade-off between confirmation bias and high uncertainty rates in sequential hypothesis testing, where conventional methods either couple stopping rules with decision criteria—introducing bias—or fully decouple them, yielding excessive inconclusive outcomes. To reconcile reliability and decisiveness, the authors propose DPitG, a novel Bayesian approach that jointly incorporates a precision target (posterior highest density interval [HDI] width ≤ ω) and a definitive decision rule (HDI entirely within or outside the region of practical equivalence [ROPE]) into its stopping criterion. Built upon the HDI-ROPE framework, DPitG accommodates both binary and continuous data and provides a closed-form sample size planning formula. In simulations of a fair coin test, DPitG reduces the rate of uncertain conclusions from 62% to 2% with only a 5% increase in median sample size, achieves zero false positives, and maintains conclusion certainty above 97%, substantially outperforming existing methods.
📝 Abstract
Sequential hypothesis testing offers flexibility over fixed-sample designs, but stopping rules coupled to decision criteria risk confirmation bias through early peeking. The HDI+ROPE algorithm exemplifies this: it stops as soon as the posterior Highest Density Interval (HDI) falls entirely outside a Region of Practical Equivalence (ROPE) around the null (rejecting it) or entirely inside, which enables positive acceptance of the null, something Null Hypothesis Significance Testing cannot do. However, this coupling means a wide HDI can satisfy the criterion on early, unrepresentative samples, incurring a systematic false-positive cost. Fully decoupled methods, such as ``Precision is the Goal'' (PitG), eliminate this bias by halting only once a target HDI width $ω$ is reached; yet, by divorcing the stopping rule from the decision criterion, PitG frequently yields inconclusive outcomes, particularly when the null hypothesis is true. We propose ``Decisive Precision is the Goal'' (DPitG), which requires both the precision target and a conclusive verdict to be satisfied simultaneously. In fair coin simulations ($ω=0.08$, ROPE$=0.5\pm0.05$), DPitG reduces the PitG inconclusive rate from 62% to 2% at a median cost of only 5% more samples, with zero false positives; HDI+ROPE achieves comparable conclusiveness only at the cost of false positives. Across the $ω$ range tested, DPitG conclusiveness remains above 97% while PitG's varies widely. We provide a closed-form planning formula $N\propto V/ω^2$ (where $V$ is the observation variance), an online interactive calculator, and open-source code. Demonstrated on single-group binary data, the framework extends readily to continuous outcomes and two-group comparisons. DPitG's fully pre-specified stopping rule is compatible with pre-registration standards and is the method of choice whenever a reliable verdict is required.