Preregistration does not improve the transparent evaluation of severity in Popper's philosophy of science or when deviations are allowed

2024-08-22

AI Summary

This study examines whether preregistration enhances the severity—i.e., the capacity to detect erroneous interpretations—of hypothesis testing, within Popperian falsificationism and Mayo’s error-statistical framework. Method: Through conceptual analysis, philosophical logic, and error-statistical modeling, the paper rigorously evaluates preregistration’s theoretical and practical implications for severity assessment. Contribution/Results: It demonstrates, for the first time, that preregistration fails to strengthen severity in the Popperian paradigm, as it does not alter a theory’s inherent falsifiability. Moreover, under realistic “planning-not-prison” practices—where deviations from preregistered protocols are permitted—the Type I error rate becomes uncontrolled, undermining Mayo’s severity criterion. Consequently, preregistration neither fortifies falsificationist logic nor ensures inferential reliability. These findings challenge the prevailing methodological consensus on preregistration’s epistemic superiority and provide foundational philosophical and statistical grounds for reevaluating scientific practice and metascientific policy.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Game Theory and Economic Paradigms: Imperfect Information
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

One justification for preregistering research hypotheses, methods, and analyses is that it improves the transparent evaluation of the severity of hypothesis tests. In this article, I consider two cases in which preregistration does not improve this evaluation. First, I argue that, although preregistration may facilitate the transparent evaluation of severity in Mayo's error statistical philosophy of science, it does not facilitate this evaluation in Popper's theory-centric approach. To illustrate, I show that associated concerns about Type I error rate inflation are only relevant in the error statistical approach and not in a theory-centric approach. Second, I argue that a test procedure that is preregistered but that also allows deviations in its implementation (i.e.,"a plan, not a prison") does not provide a more transparent evaluation of Mayoian severity than a non-preregistered procedure. In particular, I argue that sample-based validity-enhancing deviations cause an unknown inflation of the test procedure's Type I error rate and, consequently, an unknown reduction in its capability to license inferences severely. I conclude that preregistration does not improve the transparent evaluation of severity (a) in Popper's philosophy of science or (b) in Mayo's approach when deviations are allowed.

Problem

• Preregistration fails to enhance severity evaluation in Popper's theory-centric approach
• Deviations in preregistered tests obscure Type I error rate transparency
• Preregistration doesn't improve severity assessment when flexible implementations are permitted

Innovation

• Preregistration fails in Popper's theory-centric approach
• Deviation-allowed preregistration lacks severity transparency
• Type I error inflation unknown in flexible procedures