KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models

2025-08-21

AI Summary

Evaluation of knowledge graph completion (KGC) models often yields inconsistent rankings across datasets and metrics, hindering reliable cross-scenario comparisons. To address this, we propose KG-EDAS, the first unified meta-evaluation framework for KGC that introduces EDAS (Evaluation Based on Distance from Average Solution) into this domain. KG-EDAS normalizes and aggregates diverse metrics—including MRR, MR, and Hit@k—into a single, interpretable composite score, validated on benchmarks such as FB15k-237 and WN18RR. By mitigating metric-induced ranking conflicts, it significantly improves consistency, robustness, and generalizability of model rankings. Extensive experiments demonstrate that KG-EDAS produces stable, reliable ordinal relationships among state-of-the-art KGC models. As the first meta-evaluation standard for KGC with both theoretical rigor and practical feasibility, KG-EDAS establishes a principled foundation for fair, reproducible model assessment.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Knowledge Graphs (KGs) enable applications in various domains such as semantic search, recommendation systems, and natural language processing. KGs are often incomplete, missing entities and relations, an issue addressed by Knowledge Graph Completion (KGC) methods that predict missing elements. Different evaluation metrics, such as Mean Reciprocal Rank (MRR), Mean Rank (MR), and Hit@k, are commonly used to assess the performance of such KGC models. A major challenge in evaluating KGC models, however, lies in comparing their performance across multiple datasets and metrics. A model may outperform others on one dataset but underperform on another, making it difficult to determine overall superiority. Moreover, even within a single dataset, different metrics such as MRR and Hit@1 can yield conflicting rankings, where one model excels in MRR while another performs better in Hit@1, further complicating model selection for downstream tasks. These inconsistencies hinder holistic comparisons and highlight the need for a unified meta-metric that integrates performance across all metrics and datasets to enable a more reliable and interpretable evaluation framework. To address this need, we propose KG Evaluation based on Distance from Average Solution (EDAS), a robust and interpretable meta-metric that synthesizes model performance across multiple datasets and diverse evaluation criteria into a single normalized score ($M_i in [0,1]$). Unlike traditional metrics that focus on isolated aspects of performance, EDAS offers a global perspective that supports more informed model selection and promotes fairness in cross-dataset evaluation. Experimental results on benchmark datasets such as FB15k-237 and WN18RR demonstrate that EDAS effectively integrates multi-metric, multi-dataset performance into a unified ranking, offering a consistent, robust, and generalizable framework for evaluating KGC models.

Problem

• Addresses inconsistent KGC model rankings across datasets
• Resolves conflicting metric evaluations like MRR vs Hit@k
• Provides unified meta-metric for cross-dataset performance integration

Innovation

• Meta-metric framework integrating multiple evaluation metrics
• Normalized score synthesizing performance across datasets
• Distance-based approach enabling unified model comparison