kNN Algorithm for Conditional Mean and Variance Estimation with Automated Uncertainty Quantification and Variable Selection

2024-02-02

arXiv.org

AI Summary

This paper addresses the challenge of conditional distribution reconstruction under high-dimensional covariates. We propose a novel k-nearest neighbors (kNN) semi-parametric regression method that jointly estimates the conditional mean and variance to reconstruct the conditional density function and quantify predictive uncertainty. Our key contributions are: (1) the first semi-parametric ROC curve estimation framework within kNN; (2) a theoretically grounded, adaptive k-selection algorithm; and (3) a conditional distribution modeling framework integrated with variable selection. Under low-dimensional structural assumptions—such as intrinsic dimensionality or sparsity—we establish consistency and derive the optimal nonparametric convergence rate. Simulation studies demonstrate substantial improvements over conventional kNN methods in estimation accuracy and uncertainty calibration. Furthermore, two real-world biomedical applications confirm the method’s robustness and practical utility in high-dimensional settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Graphical Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we introduce a kNN-based regression method that synergizes the scalability and adaptability of traditional non-parametric kNN models with a novel variable selection technique. This method focuses on accurately estimating the conditional mean and variance of random response variables, thereby effectively characterizing conditional distributions across diverse scenarios.Our approach incorporates a robust uncertainty quantification mechanism, leveraging our prior estimation work on conditional mean and variance. The employment of kNN ensures scalable computational efficiency in predicting intervals and statistical accuracy in line with optimal non-parametric rates. Additionally, we introduce a new kNN semi-parametric algorithm for estimating ROC curves, accounting for covariates. For selecting the smoothing parameter k, we propose an algorithm with theoretical guarantees.Incorporation of variable selection enhances the performance of the method significantly over conventional kNN techniques in various modeling tasks. We validate the approach through simulations in low, moderate, and high-dimensional covariate spaces. The algorithm's effectiveness is particularly notable in biomedical applications as demonstrated in two case studies. Concluding with a theoretical analysis, we highlight the consistency and convergence rate of our method over traditional kNN models, particularly when the underlying regression model takes values in a low-dimensional space.

Problem

• Estimating conditional mean and variance via k-NN regression
• Automating variance selection to improve empirical performance
• Reconstructing conditional distributions for generative models

Innovation

• k-NN regression for joint mean and variance estimation
• Automated data-driven variable selection for improved performance
• Practical smoothing parameter rules for finite sample precision