Testing sign congruence between two parameters

2024-05-20

AI Summary

This paper addresses the two-parameter sign-consistency testing problem—testing (H_0: ext{sign}(mu_1) = ext{sign}(mu_2))—with applications in heterogeneous treatment effects, mediation analysis, reduced-form causal inference, and meta-research. To overcome the low statistical power of existing methods, we propose two new tests: (i) a practical, conservative test designed to be easily rejected under the null; and (ii) an optimal, unbiased test with exact Type I error control, grounded in boundary hypothesis testing theory—the first systematic characterization of the fundamental bias–unbiasedness trade-off in sign testing. Both methods leverage the asymptotically normal joint distribution of estimators, ensuring theoretical rigor and computational simplicity. Empirical applications demonstrate substantial gains in statistical significance for key estimates from Kowalski (2022) and Dippel et al. (2021), validating the method’s effectiveness and practical utility in empirical research.

Technology Category

• Reasoning under Uncertainty: Causality
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We test the null hypothesis that two parameters $(mu_1,mu_2)$ have the same sign, assuming that (asymptotically) normal estimators $(hat{mu}_1,hat{mu}_2)$ are available. Examples of this problem include the analysis of heterogeneous treatment effects, causal interpretation of reduced-form estimands, meta-studies, and mediation analysis. A number of tests were recently proposed. We recommend a test that is simple and rejects more often than many of these recent proposals. Like all other tests in the literature, it is conservative if the truth is near $(0,0)$ and therefore also biased. To clarify whether these features are avoidable, we also provide a test that is unbiased and has exact size control on the boundary of the null hypothesis, but which has counterintuitive properties and hence we do not recommend. We use the test to improve p-values in Kowlaksi (2022) from information contained in that paper's main text and to establish statistical significance of some key estimates in Dippel et al. (2021).

Problem

• Tests if two parameters share the same sign
• Compares asymptotic normal estimators of parameters
• Addresses conservatism and bias near null hypothesis boundary

Innovation

• Tests same sign hypothesis for two parameters
• Recommends simple, more rejecting test
• Provides unbiased test with exact size control