State-Space Models for Tabular Prior-Data Fitted Networks

2025-10-16

AI Summary

To address the quadratic computational complexity of Transformers in TabPFN with respect to sequence length and the order sensitivity of state space models (SSMs)—which renders them unsuitable for unordered tabular data—this paper proposes TabSSM, an efficient foundation model for tabular data built upon a bidirectional Hydra-structured SSM. TabSSM employs bidirectional linear temporal modeling to achieve symmetric context aggregation, thereby mitigating SSMs’ inherent sequential dependency. It integrates pretraining with approximate Bayesian inference to enhance robustness against row-order perturbations while preserving O(L) time complexity. Experiments demonstrate that TabSSM matches TabPFN’s predictive performance while significantly improving scalability with sequence length. Thus, TabSSM establishes a new paradigm for tabular modeling that jointly ensures computational efficiency, permutation robustness, and theoretical consistency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advancements in foundation models for tabular data, such as TabPFN, demonstrated that pretrained Transformer architectures can approximate Bayesian inference with high predictive performance. However, Transformers suffer from quadratic complexity with respect to sequence length, motivating the exploration of more efficient sequence models. In this work, we investigate the potential of using Hydra, a bidirectional linear-time structured state space model (SSM), as an alternative to Transformers in TabPFN. A key challenge lies in SSM's inherent sensitivity to the order of input tokens - an undesirable property for tabular datasets where the row order is semantically meaningless. We investigate to what extent a bidirectional approach can preserve efficiency and enable symmetric context aggregation. Our experiments show that this approach reduces the order-dependence, achieving predictive performance competitive to the original TabPFN model.

Problem

• Addressing quadratic complexity of Transformers in tabular data models
• Mitigating SSM sensitivity to input order in tabular datasets
• Enabling efficient bidirectional context aggregation for tabular inference

Innovation

• Uses Hydra bidirectional state-space model
• Replaces Transformers for efficient tabular processing
• Reduces order-dependence while maintaining performance