Do Tabular Foundation Models Agree with Themselves?

2026-08-06

AI Summary

This work investigates whether state-of-the-art tabular foundation models (TFMs) adhere to fundamental mathematical properties of joint distributions when performing autoregressive multivariate forecasting. To this end, we introduce two consistency criteria—marginalization consistency and factorization consistency—and develop the first theoretical framework capable of evaluating the internal consistency of TFMs without access to ground-truth posteriors. Empirical evaluations of Transformer-based TFMs across multiple classification and regression datasets reveal that all tested models violate both criteria, producing predictions that cannot correspond to any valid joint distribution. These findings expose a fundamental flaw in current approaches to multivariate prediction with TFMs, highlighting a critical gap between their practical deployment and probabilistic coherence.

Abstract

Tabular Foundation Models (TFMs) are currently the best approach to tabular prediction problems. They are constructed as transformers that approximate the Bayesian posterior predictive distribution based on a pre-training prior. These univariate predictors can be converted into multivariate ones autoregressively by sampling one target and adding it to the features. However, the faithfulness of the resulting joint has not been investigated. Furthermore, TFMs cannot be evaluated against the posterior itself, at least not on real-world datasets, because the ground-truth distribution is unknown. We therefore propose asking a different question: could a model's predictions result from any joint distribution? To answer this question, we pose two requirements that any such model must satisfy. The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals. The second is factorization consistency, which demands that different factorization orders result in equal joint distributions. Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets.

Problem

• Tabular Foundation Models
• marginalization consistency
• factorization consistency
• joint distribution
• autoregressive prediction

Innovation

• Tabular Foundation Models
• marginalization consistency
• factorization consistency
• autoregressive sampling
• joint distribution consistency