Speedrunning Tabular Foundation Model Pretraining

2026-06-02

AI Summary

This work addresses the high computational cost, long iteration cycles, and lack of standardized evaluation protocols in pretraining foundation models for tabular data by introducing the first community challenge structured as a “speedrun.” Built upon the nanoTabPFN single-file script, participants compete to reach a fixed ROC AUC target on a single NVIDIA L40S GPU. The framework standardizes research on training acceleration, enabling method composition, validation, and sharing, while maintaining a public leaderboard. By integrating data subsampling and optimized training strategies, the approach achieves a new state-of-the-art pretraining time of 0.92 minutes on the TabArena benchmark—81× faster than the 74.32-minute baseline—and reduces synthetic data usage by 22×.

Abstract

Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimization ideas. Yet the community lacks a simple way to compare and accumulate pretraining speedups. We introduce a community speedrun for nanoTabPFN: contributors modify a single-file training script and compete to reach a fixed downstream ROC AUC target on subsampled TabArena using one NVIDIA L40S GPU. The current best record reaches the target in 0.92 minutes, an 81x speedup over the 74.32 minute baseline while using 22x fewer synthetic datasets. The speedrun format provides a simple protocol for the community to add, verify, and stack pretraining improvements, with the leaderboard open to contributions. Code and records are available at https://github.com/borawhocodess/modded-nanotabpfn.

Problem

• tabular foundation models
• pretraining cost
• speedup
• research bottleneck
• iteration cycle

Innovation

• tabular foundation models
• pretraining acceleration
• community speedrun
• nanoTabPFN
• efficient training