Mambular: A Sequential Model for Tabular Deep Learning

2024-08-12

arXiv.org

AI Summary

Tabular data modeling has long been dominated by gradient-boosted decision trees (GBDTs), while deep learning models remain poorly adapted to structured tabular inputs. Method: This work introduces structured state space models (SSMs)—specifically Mamba—into tabular learning for the first time, proposing an autoregressive tabular modeling framework. It treats features as sequences and incorporates feature embedding, learnable positional encoding, bidirectional state space modeling, feature interaction modules, and adaptive pooling. Contribution/Results: Extensive experiments across diverse benchmark datasets demonstrate performance competitive with state-of-the-art GBDTs and advanced deep models. This work establishes a novel paradigm for applying SSMs to tabular data and releases an open-source implementation to advance deep learning for tabular data.

Technology Category

• Machine Learning: Structured Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We extensively benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods, and demonstrate its competitive performance across diverse datasets. Additionally, we explore various adaptations of Mambular to understand its effectiveness for tabular data. We investigate different pooling strategies, feature interaction mechanisms, and bi-directional processing. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models. The results highlight Mambulars potential as a versatile and powerful architecture for tabular data analysis, expanding the scope of deep learning applications in this domain. The source code is available at https://github.com/basf/mamba-tabular.

Problem

• Investigates autoregressive models for tabular data performance
• Explores pooling and feature interaction adaptations
• Proposes sequential processing for improved tabular analysis

Innovation

• Autoregressive state-space models for tabular data
• Structured state-space layers process feature sequences
• Bi-directional processing enhances feature interactions