🤖 AI Summary
Traditional meta-analyses struggle to quantify the strength of evidence for the presence or absence of an effect and cannot adequately assess the sensitivity of conclusions to publication bias or small-study effects. This work proposes a Bayesian evidence auditing framework tailored to meta-analytic corpora, integrating bias-aware models with unbiased baseline specifications through Bayesian model averaging and Bayes factors. It introduces “rigor” as a composite metric that jointly evaluates the strength of evidence for an effect and robustness to bias—allowing null effects to achieve high rigor scores when supported by strong evidence. Built upon Bayesian random-effects models, the approach employs simulation and resampling strategies within the ADEMP framework, including synthetic data generation, registered-report resampling, and contour-enhanced funnel weighting. Applied to nutritional intervention studies, the method frequently attenuates conventional effect estimates, revealing that many nominally significant findings lack robust evidential support. Full reproducible resources are publicly released.
📝 Abstract
Conventional meta-analysis summarizes evidence through pooled estimates, intervals, and p-values, but these outputs do not directly measure evidence for an effect, evidence for no effect, or the degree to which conclusions depend on publication selection or small-study effects. We introduce a corpus-scale Bayesian evidential-audit workflow for meta-analytic corpora. The workflow reconstructs or accepts study-level effects and standard errors, harmonizes directions, fits a matched Bayesian random-effects baseline and a bias-aware model-averaged ensemble, and reports paired estimates with component and joint model-family evidence. The central estimand is rigor: a joint Bayes-factor summary combining resolved effect/no-effect evidence with absence of an explicit bias component in the fitted ensemble. Rigor is not a positive-finding score; no-effect evidence can score highly, whereas inconclusive or bias-dependent evidence scores poorly. We characterize the workflow using an ADEMP-framed simulation/resampling design with known-cell synthetic simulation, empirical registry resampling, and empirical fitted-profile-weighted synthetic sampling. A nutrition intervention corpus provides the worked case study, where bias-aware fitting often attenuates conventional estimates and many nominally meaningful effects lose clean evidential support. A public companion repository provides empirical inputs, generated artifacts, simulation source/design files, and documentation for reproducing and adapting the audit.