LEDD: Large Language Model-Empowered Data Discovery in Data Lakes

2025-02-21

AI Summary

Addressing the semantic discovery challenge posed by massive heterogeneous tables in data lakes—particularly natural language–driven semantic table retrieval and hierarchical global catalog generation—this paper proposes the first end-to-end framework integrating large language models (LLMs). The framework unifies semantic search and hierarchical metadata generation, synergistically combining LLMs, semantic embeddings, and hierarchical metadata modeling to enable interpretable catalog construction and high-relevance table retrieval. Implemented via a lightweight Python interface, it supports plug-and-play algorithmic modules. Experiments demonstrate significant improvements in data preparation efficiency for downstream tasks such as text-to-SQL and feature engineering. The approach establishes a scalable, reusable paradigm for semantic governance in modern data lakes.

Technology Category

Natural Language Processing: (Large) Language ModelsData Mining & Knowledge Management: Semantic WebMachine Learning: Large Multimodal Models (LMMs)

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Large language models for searchUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Data discovery in data lakes with ever increasing datasets has long been recognized as a big challenge in the realm of data management, especially for semantic search of and hierarchical global catalog generation of tables. While large language models (LLMs) facilitate the processing of data semantics, challenges remain in architecting an end-to-end system that comprehensively exploits LLMs for the two semantics-related tasks. In this demo, we propose LEDD, an end-to-end system with an extensible architecture that leverages LLMs to provide hierarchical global catalogs with semantic meanings and semantic table search for data lakes. Specifically, LEDD can return semantically related tables based on natural-language specification. These features make LEDD an ideal foundation for downstream tasks such as model training and schema linking for text-to-SQL tasks. LEDD also provides a simple Python interface to facilitate the extension and the replacement of data discovery algorithms.

Problem

• Facilitates semantic search in data lakes
• Generates hierarchical global catalogs for tables
• Leverages LLMs for end-to-end data discovery

Innovation

• LLM-empowered hierarchical global catalogs
• Semantic table search in data lakes
• Extensible Python interface for algorithms