TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

2026-06-01

AI Summary

This work addresses a critical oversight in modern tabular learning benchmarks—the neglect of feature engineering—which introduces bias in model evaluation and obscures its pivotal role in real-world applications. To bridge this gap, the authors propose TabPrep, a lightweight, domain-knowledge-driven preprocessing pipeline that systematically enhances model performance by identifying and transforming three fundamental structural data patterns through targeted feature generators. TabPrep integrates systematic feature engineering into mainstream tabular benchmarking for the first time, demonstrating that well-designed preprocessing alone can surpass performance gains from most architectural innovations. Evaluated on the TabArena benchmark, TabPrep consistently boosts the accuracy of diverse models—including tree-based methods, neural networks, linear models, and foundation models—while maintaining low computational overhead and strong generalizability.

Abstract

Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.

Problem

• tabular machine learning
• feature engineering
• benchmarks
• evaluation gap
• structural data patterns

Innovation

• feature engineering
• tabular data
• preprocessing pipeline
• model benchmarking
• structural patterns