Why Large Language Models Fail at Tabular Prediction

2026-08-03

AI Summary

This study investigates the fundamental reasons behind the poor performance of large language models (LLMs) on tabular prediction tasks under zero-shot settings without fine-tuning or external tools. By systematically evaluating five hypotheses across 31 benchmark datasets, integrating random linear projections with grid consistency analysis, and comparing against 252 classical model configurations, the work identifies dimensionality growth as the primary factor driving significant LLM performance degradation. The findings reveal that while LLMs exhibit behavior akin to local distance-based methods in low-dimensional spaces (e.g., two dimensions), they display a distinct and irreproducible degeneration pattern in high-dimensional settings—one that cannot be emulated by any classical model. In contrast, traditional methods maintain stable or even improved performance as dimensionality increases.

Abstract

Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.

Problem

• Large Language Models
• Tabular Prediction
• Dimensionality
• Predictive Analytics
• Model Failure

Innovation

• tabular data
• large language models
• dimensionality
• predictive analytics
• model failure