What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates

2026-09-23

AI Summary

研究探讨表格基础模型如何通过注意力门控更新在情境中进行表征优化，以适应每个表格定义的新监督任务，提出RefineICL方法并展示其在多个数据集上的优越性能。

Abstract

What reusable computation should a tabular foundation model learn when every table defines a new supervised task? We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes. Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.

Problem

• tabular foundation model
• supervised task
• representation refinement

Innovation

• in-situ representation refinement
• attention-gated updates
• RefineICL
• tabular foundation models
• contextual stack