LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains

2025-11-08

AI Summary

Cross-domain tabular data transfer faces dual challenges: feature-space heterogeneity and limited knowledge transfer capability of conventional models; meanwhile, large language models (LLMs) struggle to effectively model structured, mixed-type tabular data due to their reliance on textual prompting and in-context learning. To address this, we propose LLM-AT (LLM Attention Transfer), a lightweight framework introducing the first attention weight transfer mechanism for tabular transfer learning. Specifically, it transfers selectively learned key-value projection weights from a fine-tuned LLM on the source domain to a table-specific gated Feature Tokenization Transformer (gFTT), enabling cross-domain knowledge transfer without feature alignment, prompt engineering, or shared input spaces. Evaluated on ten cross-domain benchmarks, LLM-AT significantly outperforms twelve baselines—including classical machine learning methods, state-of-the-art deep tabular models, and large-scale pretrained transfer models—demonstrating superior generalization, especially under low-resource conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Transfer learning of tabular data is non-trivial due to heterogeneity in the feature space across disparate domains. The limited success of traditional deep learning in tabular knowledge transfer can be advanced by leveraging large language models (LLMs). However, the efficacy of LLMs often stagnates for mixed data types structured in tables due to the limitations of text prompts and in-context learning. We propose a lightweight transfer learning framework that fine-tunes an LLM using source tabular data and transplants the LLM's selective $key$ and $value$ projection weights into a gated feature tokenized transformer (gFTT) built for tabular data. The gFTT model with cross-domain attention is fine-tuned using target tabular data for transfer learning, eliminating the need for shared features, LLM prompt engineering, and large-scale pretrained models. Our experiments using ten pairs of source-target data sets and 12 baselines demonstrate the superiority of the proposed LLM-attention transplant for transfer learning (LATTLE) method over traditional ML models, state-of-the-art deep tabular architectures, and transfer learning models trained on thousands to billions of tabular samples. The proposed attention transfer demonstrates an effective solution to learning relationships between data tables using an LLM in a low-resource learning environment. The source code for the proposed method is publicly available.

Problem

• Transfers tabular knowledge across domains with heterogeneous feature spaces
• Overcomes LLM limitations for mixed data types in tabular structures
• Enables low-resource transfer learning without shared features or large models

Innovation

• LLM attention transplant for cross-domain transfer learning
• Gated feature tokenized transformer for tabular data
• Fine-tuning with selective key-value projection weights