TabDPT: Scaling Tabular Foundation Models
Tabular data exhibit strong heterogeneity, and existing models suffer from poor generalization and difficulty in zero-shot adaptation to new tasks. Method: We propose the Discriminative Tabular Pre-trained Transformer (TabDPT), the first framework integrating real-table-driven self-supervised pretraining with retrieval-augmented in-context learning (ICL). It introduces numerical-aware embedding and attention mechanisms, alongside a lightweight discriminative architecture. Contribution/Results: TabDPT achieves true zero-shot cross-task generalization without fine-tuning—overcoming a key bottleneck in large language models’ handling of structured numerical tables. It attains state-of-the-art zero-shot performance on the CC18 classification and CTR23 regression benchmarks. Performance scales consistently with both model and data size, while maintaining efficient inference and strong scalability.