SyNPar: Synthetic Null Data Parallelism for High-Power False Discovery Rate Control in High-Dimensional Variable Selection

2025-01-09

AI Summary

In high-dimensional variable selection, controlling the false discovery rate (FDR) while maintaining statistical power remains challenging: existing approaches—such as data splitting and knockoff methods—introduce data perturbation or partitioning, substantially compromising power. This paper proposes SyNPar, a novel “synthetic null parallel estimation” framework. SyNPar first fits a model on the original data, then generates synthetic null data that strictly satisfy the null hypothesis; coefficients are estimated in parallel on both the observed and synthetic null data, enabling false positive identification via comparative inference. Crucially, SyNPar avoids data perturbation or splitting. It is the first method to achieve asymptotically full power at any prespecified FDR level, with stronger theoretical guarantees than state-of-the-art alternatives. The framework applies broadly—including to generalized linear models, Cox proportional hazards models, and Gaussian graphical models. Extensive simulations and real-data analyses demonstrate superior FDR calibration, significantly higher power, and improved computational efficiency.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Mixed Discrete/Continuous Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Balancing false discovery rate (FDR) and statistical power to ensure reliable discoveries is a key challenge in high-dimensional variable selection. Although several FDR control methods have been proposed, most involve perturbing the original data, either by concatenating knockoff variables or splitting the data into two halves, both of which can lead to a loss of power. In this paper, we introduce a novel approach called Synthetic Null Parallelism (SyNPar), which controls the FDR in high-dimensional variable selection while preserving the original data. SyNPar generates synthetic null data from a model fitted to the original data and modified to reflect the null hypothesis. It then applies the same estimation procedure in parallel to both the original and synthetic null data to estimate coefficients that indicate feature importance. By comparing the coefficients estimated from the null data with those from the original data, SyNPar effectively identifies false positives, functioning as a numerical analog of a likelihood ratio test. We provide theoretical guarantees for FDR control at any desired level while ensuring that the power approaches one with high probability asymptotically. SyNPar is straightforward to implement and can be applied to a wide range of statistical models, including high-dimensional linear regression, generalized linear models, Cox models, and Gaussian graphical models. Through extensive simulations and real data applications, we demonstrate that SyNPar outperforms state-of-the-art methods, including knockoffs and data-splitting methods, in terms of FDR control, power, and computational efficiency.

Problem

• Controls FDR in high-dimensional variable selection
• Avoids data degradation and power loss
• Applicable to diverse high-dimensional models

Innovation

• Generates synthetic null data without altering original data
• Uses parallel estimation on original and synthetic data
• Applies data-driven correction for FDR control