Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners

2026-07-28

AI Summary

This work addresses the silent failure of in-context learning (ICL) on tabular data in novel environments, where composite features entangle causal and spurious signals. We formally characterize this issue as “spurious routing in composite representations” and introduce a closed-form metric, Composite Spuriousness Ratio (CSR), revealing that larger context sizes paradoxically exacerbate the problem. Guided by theoretical analysis and empirical validation using TabPFN, we propose a lightweight mitigation strategy requiring only weak environment labels: S-swap data augmentation combined with environment-stratified context construction. This approach reduces spurious routing by 74% for linear ICL and by 98.8% for TabPFN, while enhancing TabPFN’s causal sensitivity by 8.4×. Experimental results align closely with CSR-based theoretical predictions (r > 0.97).

Abstract

Consider a model trained at a single hospital to predict patient recovery, where the measured feature $X$ bundles the patient's true health signal ($C$) with a systematic artefact from that hospital's equipment ($S$). Within that hospital, the artefact correlates with outcomes through unmeasured confounders such as patient demographics; an in-context learner rationally routes predictions through $S$, not $C$, and fails silently when deployed at a new hospital with different equipment. We formalise this as \emph{spurious routing in composite representations}: when a feature $X = [C;\,αS;\,η]$ encodes a causal signal $C$ and a spurious signal $S$ in distinct subspaces, the ICL cannot determine which drives predictions. We prove that under ridge ICL, a linear in-context learner, this routing is unavoidable regardless of context size; TabPFN, a state-of-the-art pretrained tabular ICL model, shows qualitatively consistent behaviour empirically. We derive a closed-form characterisation, $\mathrm{CSR} \propto ρ_S/ρ_C$, confirmed at $r = 0.997$ for linear ICL and $r = 0.979$ for TabPFN. Contrary to intuition, larger context sharpens commitment to the dominant in-context signal, amplifying spurious routing by up to $1.74\times$; in the high-spurious corner, more expressive models show greater vulnerability empirically ($+2.22$ CSR gap at high entanglement). We introduce two lightweight mitigations: environment-stratified context construction and S-swap augmentation, that require only weak environment labels and no knowledge of the causal partition. S-swap reduces spurious routing by $74\%$ for linear ICL and $98.8\%$ for TabPFN, with TabPFN's causal sensitivity increasing $8.4\times$ simultaneously: the model does not become agnostic, it reroutes through the causal signal.

Problem

• spurious routing
• in-context learning
• tabular data
• causal signal
• distribution shift

Innovation

• spurious routing
• in-context learning
• composite representations
• causal signal recovery
• tabular foundation models