TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

2026-09-29

AI Summary

This study addresses the limitations of existing tabular foundation models, which lack semantic understanding of data, and the high noise and overfitting susceptibility inherent in conventional self-evolving agent-based search. To overcome these challenges, this work proposes a self-evolving machine learning engineering framework driven by large language model agents, pioneering the application of self-evolving pipelines to frozen tabular foundation models. By iteratively optimizing data cleaning, feature engineering, and post-processing procedures through the integration of metadata and feedback signals, the proposed approach achieves efficient cross-dataset transferability and semantic enhancement. Empirical evaluations demonstrate that this method secures the top five positions on the TabArena benchmark with substantial improvements in Elo ratings, while also achieving first place in the MLE-Bench competition.

Abstract

Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM's error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. The discovered pipelines also transfer to other frozen tabular foundation models (+69 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.

Problem

• Tabular Foundation Models
• Self-Evolving Pipelines
• Machine Learning Engineering Agents
• Dataset Semantics
• Overfitting

Innovation

• Tabular Foundation Models
• Self-Evolving Pipelines
• LLM Agents
• Feature Engineering
• Zero-Shot Transfer