Robust Ultra-High-Dimensional Variable Selection With Correlated Structure Using Group Testing

2026-02-06

AI Summary

High-dimensional genomic data often exhibit strong within-group correlations, yet conventional variable selection methods are limited by their reliance on independence assumptions, prior pathway knowledge, or sensitivity to outliers. This work proposes a robust Dorfman screening framework that constructs data-driven variable groups via hierarchical clustering and integrates group- and within-group hypothesis testing, followed by refined selection using elastic net or adaptive elastic net regularization. The approach incorporates robust techniques—including OGK covariance estimation, rank-based correlation, and Huber-weighted regression—to effectively handle non-normal or contaminated data without requiring prior pathway information. In both simulation studies and real-world NSCLC gene expression data, the method substantially outperforms existing approaches, achieving the lowest prediction error and successfully enriching clinically relevant genes.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Search and Optimization: Mixed Discrete/Continuous Search
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Background: High-dimensional genomic data exhibit strong group correlation structures that challenge conventional feature selection methods, which often assume feature independence or rely on pre-defined pathways and are sensitive to outliers and model misspecification. Methods: We propose the Dorfman screening framework, a multi-stage procedure that forms data-driven variable groups via hierarchical clustering, performs group and within-group hypothesis testing, and refines selection using elastic net or adaptive elastic net. Robust variants incorporate OGK-based covariance estimation, rank-based correlation, and Huber-weighted regression to handle contaminated and non-normal data. Results: In simulations, Dorfman-Sparse-Adaptive-EN performed best under normal conditions, while Robust-OGK-Dorfman-Adaptive-EN showed clear advantages under data contamination, outperforming classical Dorfman and competing methods. Applied to NSCLC gene expression data for trametinib response, robust Dorfman methods achieved the lowest prediction errors and enriched recovery of clinically relevant genes. Conclusions: The Dorfman framework provides an efficient and robust approach to genomic feature selection. Robust-OGK-Dorfman-Adaptive-EN offers strong performance under both ideal and contaminated conditions and scales to ultra-high-dimensional settings, making it well suited for modern genomic biomarker discovery.

Problem

• high-dimensional data
• group correlation
• feature selection
• outliers
• model misspecification

Innovation

• Dorfman screening
• robust variable selection
• group testing
• ultra-high-dimensional data
• adaptive elastic net