Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Design

2026-03-18

AI Summary

This study addresses the lack of systematic design guidance for large language model (LLM)-based embedding pipelines in tabular prediction tasks. It presents the first large-scale empirical investigation, systematically evaluating 256 pipeline configurations that combine eight preprocessing strategies, sixteen embedding models, and two downstream models—including gradient-boosted decision trees. The findings reveal that embedding concatenation significantly outperforms column-wise replacement, that larger embedding models consistently yield greater performance gains than those selected solely by popularity, and that integrating gradient-boosted decision trees further enhances predictive accuracy. These results establish reliable design principles and practical guidelines for effectively leveraging LLM-derived embeddings in tabular data prediction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Embeddings are a powerful way to enrich data-driven machine learning models with the world knowledge of large language models (LLMs). Yet, there is limited evidence on how to design effective LLM-based embedding pipelines for tabular prediction. In this work, we systematically benchmark 256 pipeline configurations, covering 8 preprocessing strategies, 16 embedding models, and 2 downstream models. Our results show that it strongly depends on the specific pipeline design whether incorporating the prior knowledge of LLMs improves the predictive performance. In general, concatenating embeddings tends to outperform replacing the original columns with embeddings. Larger embedding models tend to yield better results, while public leaderboard rankings and model popularity are poor performance indicators. Finally, gradient boosting decision trees tend to be strong downstream models. Our findings provide researchers and practitioners with guidance for building more effective embedding pipelines for tabular prediction tasks.

Problem

• embedding pipeline
• tabular prediction
• world knowledge
• LLM embeddings
• pipeline design

Innovation

• embedding pipelines
• tabular data
• large language models
• systematic benchmarking
• gradient boosting