Enhancing Tabular Learners with Context-Aware Semantic Embeddings

2026-08-04

AI Summary

This work addresses the limitations of existing tabular learning methods, which treat feature names and cell values as discrete symbols and thus overlook their semantic content, leading to degraded performance under data sparsity. To overcome this, the authors propose CASE, a framework that integrates the semantic comprehension of large language models with the statistical modeling capacity of tabular learners through a context-aware mechanism for row embedding generation. The core innovation lies in a customized Gemma-3-based tabular language model augmented with a pre-filled KV cache strategy, which leverages representative samples as semantic anchors to produce dynamically contextualized embeddings, effectively mitigating semantic ambiguity. Experimental results demonstrate that CASE achieves substantial performance gains across multiple benchmarks—including CARTE, TextTab, and TabArena—with particularly pronounced advantages in low-resource settings.

Abstract

While modern tabular learners excel at capturing statistical patterns, they frequently operate in a semantic vacuum, treating textual features as discrete symbols, ignoring the rich semantics inherent in feature names or cell entries. We propose CASE (Context-Aware Semantic Embeddings), a novel framework that bridges the gap between the semantic understanding of Large Language Models (LLMs) and the statistical capabilities of tabular learners. Unlike existing methods that embed rows in isolation, CASE utilizes a contextualization strategy: we pre-fill the KV cache of a custom-trained Gemma 3-based Tabular Language Model with a representative sample of rows to establish a persistent anchor of the dataset's semantics. This ensures that generated row embeddings are dynamically contextualized, resolving semantic ambiguities and anchoring representations in domain-specific context. Our experiments across several benchmarks (CARTE, TextTab, and TabArena) demonstrate that CASE substantially improves the performance of tabular learners on semantically rich datasets, particularly in low-data regimes.

Problem

• tabular learning
• semantic embeddings
• context-awareness
• feature semantics
• LLM integration

Innovation

• Context-Aware Semantic Embeddings
• Tabular Learning
• Large Language Models
• KV Cache Contextualization
• Semantic Disambiguation