E-Values For Multiplicity Control In Multiverse Analysis

2026-07-20

AI Summary

This study addresses the inflated false positive rates in multiverse analyses caused by multiple comparisons, a problem often exacerbated by existing methods that rely on strong structural assumptions. The authors introduce, for the first time, a systematic application of e-values—including universal e-values, soft-rank e-values, and p-to-e calibration—within a generalized linear model framework to control the false discovery rate without requiring such assumptions, while also enabling adjustment for confounding covariates. Both theoretical analysis and empirical evaluation demonstrate that the proposed p-to-e calibration substantially outperforms current approaches in typical multiverse settings. Applying this method reveals robust associations between adolescent depression, low self-esteem, peer difficulties, and increased use of the internet and social media.

Abstract

Multiverse analysis refers to a common situation where one wishes to assess the association between multiple possible treatment definitions and multiple possible outcome definitions, potentially within multiple sub-populations, among other possible analysis specifications. Multiverse analysis is a useful exploratory tool to assess heterogeneity across the considered specifications, but it is sometimes also used to assess statistical significance. In the latter case, it is critical to acknowledge that multiple comparisons are being performed, and to ensure a valid statistical control of false positive findings. We study the use of e-values within generalized linear models as a tool to control the false discovery rate regardless of the dependence structure of the multiple analyses being performed, while accounting for confounding covariates. We compare the performance of several approaches: universal e-values, soft-rank e-values, and p-to-e calibration. We find that, for problem characteristics typically encountered in multiverse analyses, p-to-e calibration significantly outperforms the other two approaches in terms of statistical power, but said power may be moderate unless the sample size or effect sizes are large enough. An application studying the association between teenager technology use and mental well-being reveals association between depression, low self-steem and peer problems with internet and social media usage.

Problem

• multiplicity control
• multiverse analysis
• false discovery rate
• multiple comparisons
• e-values

Innovation

• e-values
• multiverse analysis
• false discovery rate
• multiple comparisons
• p-to-e calibration