Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models

2026-10-05

AI Summary

This study addresses the cognitive bias in evaluating active feature acquisition strategies caused by uneven coverage in offline data, which conflates uncertainty arising from missing data (epistemic) with uninformative features (aleatoric). We propose an evaluation framework based on Prior-data Fitted Networks (PFNs), whose core innovation lies in replacing total predictive entropy with posterior expected aleatoric entropy as the reward function. This effectively decouples epistemic and aleatoric uncertainties, thereby eliminating evaluation bias. By integrating tabular foundation models with active feature learning techniques, the proposed method significantly reduces value estimation bias on both synthetic and real-world datasets. Furthermore, it generates confidence intervals with high empirical coverage and substantially enhances downstream policy selection performance.

Abstract

Active feature acquisition learns policies that sequentially acquire features to maximize information about a target variable. We study how to learn and evaluate such policies from finite offline data using prior-data fitted networks (PFNs), which are off-the-shelf models that output posterior predictive distributions without task-specific training. We show that under the imbalanced coverage of offline data, using total predictive entropy as a reward creates an epistemic bias that penalizes acquiring sparsely observed features. Specifically, this reward conflates epistemic uncertainty (arising from lack of offline data) with aleatoric uncertainty (arising from uninformative features). To address this, we target the posterior expected (aleatoric) entropy instead of the total predictive entropy output by a PFN for evaluating feature acquisitions. Empirical evaluations on synthetic and real-world datasets demonstrate that our approach consistently reduces value estimation bias and yields credible intervals with strong empirical coverage, which can translate to improved downstream policy selection.

Problem

• Active Feature Acquisition
• Offline Data
• Epistemic Bias
• Predictive Entropy
• Uncertainty Disentanglement

Innovation

• Active Feature Acquisition
• Tabular Foundation Models
• Prior-data Fitted Networks
• Uncertainty Decomposition
• Offline Policy Evaluation