Hypothesis-free discovery from epidemiological data by automatic detection and local inference for tree-based nonlinearities and interactions

2025-05-01

AI Summary

Machine learning in epidemiology excels at hypothesis-free discovery of nonlinearities and interactions but lacks reliable methods for local statistical inference—particularly for quantifying uncertainty in feature effects. To address this, we propose RuleSHAP: a novel framework that tightly integrates sparse Bayesian regression, tree ensembles (XGBoost/LightGBM), and Shapley value theory into a single-step, end-to-end pipeline for interpretable, statistically testable individualized effect estimation. RuleSHAP automatically identifies and performs valid statistical inference on high-dimensional nonlinear interactions (e.g., complex dependencies of cholesterol–blood pressure associations on age, sex, BMI, and glucose). Simulation studies confirm its statistical validity—including correct coverage of confidence intervals and calibrated p-values. Applied to real-world cohort data, RuleSHAP successfully replicates established clinical mechanisms while uncovering novel, biologically plausible interactions. By bridging model-agnostic interpretability with rigorous local inference, RuleSHAP substantially enhances both the reliability and interpretability of hypothesis-free epidemiological discovery.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Data Mining & Knowledge Management: Rule Mining & Pattern Mining

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In epidemiological settings, Machine Learning (ML) is gaining popularity for hypothesis-free discovery of risk (or protective) factors. Although ML is strong at discovering non-linearities and interactions, this power is currently compromised by a lack of reliable inference. Although local measures of feature effect can be combined with tree ensembles, uncertainty quantifications for these measures remain only partially available and oftentimes unsatisfactory. We propose RuleSHAP, a framework for using rule-based, hypothesis-free discovery that combines sparse Bayesian regression, tree ensembles and Shapley values in a one-step procedure that both detects and tests complex patterns at the individual level. To ease computation, we derive a formula that computes marginal Shapley values more efficiently for our setting. We demonstrate the validity of our framework on simulated data. To illustrate, we apply our machinery to data from an epidemiological cohort to detect and infer several effects for high cholesterol and blood pressure, such as nonlinear interaction effects between features like age, sex, ethnicity, BMI and glucose level.

Problem

• Lack of reliable inference in ML for epidemiological risk factors
• Uncertainty quantification for local feature effects in tree ensembles
• Detecting and testing complex patterns at individual level efficiently

Innovation

• Combines sparse Bayesian regression with tree ensembles
• Uses Shapley values for local feature effect inference
• Efficient computation of marginal Shapley values