Transfer Learning of Tabular Data by Finetuning Large Language Models

2025-01-12

AI Summary

Deep learning models face performance bottlenecks in few-shot, low-dimensional (<10 features) tabular classification due to difficulties in modeling heterogeneous features and poor transferability. Method: This paper introduces, for the first time, an end-to-end fine-tuning paradigm for large language models (LLMs) on raw tabular data—requiring no textual conversion, external API calls, or pre-trained tabular models. It employs table serialization encoding and a lightweight adapter to enable efficient cross-dataset transfer. Results: Our approach outperforms state-of-the-art machine learning and deep learning methods across 10 benchmark datasets. It reduces computational overhead to only 20–33% of that incurred by API-based inference or full-parameter fine-tuning, while achieving higher classification accuracy. The core contribution lies in overcoming the technical limitations of applying LLMs directly to structured tabular data, establishing a low-cost, high-accuracy, plug-and-play transfer learning framework for small-scale tabular tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.

Problem

• Deep Learning
• Tabular Data
• Transfer Learning

Innovation

• Large Pre-trained Model
• Transfer Learning
• Tabular Data Classification