KcMF: A Knowledge-compliant Framework for Schema and Entity Matching with Fine-tuning-free LLMs

2024-10-16

arXiv.org

AI Summary

Large language models (LLMs) suffer from hallucination and instruction confusion in schema matching (SM) and entity matching (EM) tasks. Method: We propose KcMF, a zero-fine-tuning framework featuring (i) a novel pseudocode-driven unified task decomposition paradigm; (ii) a dual knowledge injection mechanism—“Data-as-Knowledge” (DaK) and “Example-as-Knowledge” (EaK); and (iii) a multi-LLM output fusion strategy integrated with zero-shot prompting and structured output constraints. Contribution/Results: KcMF introduces pseudocode as a universal task orchestration language, enabling cross-task and cross-model knowledge transfer via interpretable, executable representations. Evaluated on five mainstream LLMs, it achieves an average F1-score gain of 17.93% over non-LLM baselines on SM/EM benchmarks, demonstrating substantial improvements in zero-shot generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Schema matching (SM) and entity matching (EM) tasks are crucial for data integration. While large language models (LLMs) have shown promising results in these tasks, they suffer from hallucinations and confusion about task instructions. This study presents the Knowledge-Compliant Matching Framework (KcMF), an LLM-based approach that addresses these issues without the need for domain-specific fine-tuning. KcMF employs a once-and-for-all pseudo-code-based task decomposition strategy to adopt natural language statements that guide LLM reasoning and reduce confusion across various task types. We also propose two mechanisms, Dataset as Knowledge (DaK) and Example as Knowledge (EaK), to build domain knowledge sets when unstructured domain knowledge is lacking. Moreover, we introduce a result-ensemble strategy to leverage multiple knowledge sources and suppress badly formatted outputs. Extensive evaluations confirm that KcMF clearly enhances five LLM backbones in both SM and EM tasks while outperforming the non-LLM competitors by an average F1-score of 17.93%.

Problem

• Improves schema and entity matching accuracy
• Eliminates need for domain-specific fine-tuning
• Enhances LLM performance across various tasks

Innovation

• Fine-tuning-free LLM approach
• Pseudo-code-based task decomposition
• Dataset and Example as Knowledge