🤖 AI Summary
This study addresses the lack of systematic evaluation in outlier handling within meta-analyses, which can render conclusions susceptible to subjective methodological choices. We preregistered and systematically compared four commonly used outlier detection and adjustment methods—including Winsorizing and DFBETAS—across 358 meta-analyses in the behavioral sciences, employing random-effects models with unrestricted weighted least squares estimation. For the first time at scale, we quantified how these approaches influence pooled effect sizes, statistical significance, and the smallest effect size of interest. Results indicate that while outlier treatment exerts minimal impact on average effect estimates (median change ≤ 0.047), it reverses significance judgments in 11.5% of cases and alters effect size interpretations in 15.9%, particularly among marginally significant findings.
📝 Abstract
Meta-analysts routinely face estimates that look too large or extreme. Yet, how to handle them is left to the reviewer's judgment. The methods for detecting such estimates are well known. What is missing is an informed assessment of how much alternative handling choices might change a meta-analysis' conclusions. We fill this gap by analyzing the effects of four pre-registered handling treatments across 358 behavioral science meta-analyses with at least ten estimates. Each outlier handling treatment is estimated by two estimators (random effects and unrestricted weighted least squares), and compared to the 'do-nothing' baseline on three outcomes: the pooled effect, statistical significance, and whether the effect reaches the smallest effect size of interest (|d| >= 0.20). Our entire analysis and comparison pipelines were pre-registered. Alternative outlier handling treatments have little effect on the meta-analysis mean as the median absolute change in Cohen's d is at most 0.047 and often much less. Yet, at least one of these four treatments in combination with one of these estimators reverses the statistical significance of 11.5% of meta-analyses and the smallest-effect-of-interest assessment in 15.9%. Winsorizing has the least effect and DFBETAS the most. Categorical changes are found almost entirely among results already close to the decision boundary; strongly significant results essentially never change. These findings give applied meta-analysts, methods specialists, and reviewers a reference point for how much this under-reported choice matters and provide yet another reason for meta-analysts to publicly pre-specify their methods and handling treatments.