link prediction evaluation

This study addresses the inconsistency in model rankings caused by commonly used ranking metrics—such as MRR, Hits@k, and Mean Rank—in knowledge graph completion (KGC) evaluation, which hinders fair comparison and reproducibility. For the first time, KGC evaluation is framed as a multi-criteria decision-making problem, and seven aggregators are systematically assessed across five dimensions: consistency, cross-dataset stability, metric independence, noise robustness, and generalization capability. Through leave-one-model-out (LOMO) and leave-one-group-out (LOGO) cross-validation, Pareto optimality analysis, and multidimensional sensitivity tests, the Z-score aggregator emerges as the most balanced overall—favoring DualE for tail entity prediction and FMS for relation prediction. The experiments further reveal that consistency and stability are insensitive to removal strategies, whereas generalization and independence exhibit the highest sensitivity.

This work addresses the limitations of conventional knowledge graph completion (KGC) model evaluation, which predominantly relies on uniform ranking metrics and fails to accommodate diverse user assessment needs. To bridge this gap, we propose PROBE-Web—the first interactive, goal-oriented evaluation system that enables flexible and fine-grained model analysis by adjusting two key perspectives: prediction sharpness and robustness to popularity bias. The system integrates four core functionalities: standard evaluation, perspective-aware assessment, interpretable case studies, and evaluation landscape exploration. Through an intuitive graphical interface, PROBE-Web supports dynamic multi-model comparison, real-time metric computation, and interactive visualization. Empirical results demonstrate that PROBE-Web effectively uncovers performance disparities among KGC models under multidimensional evaluation criteria, substantially enhancing the transparency and practical utility of model assessment.

This paper addresses the long-standing absence of principled quality assessment criteria for graph datasets in graph learning. We propose RINGS, the first first-principles-based framework for evaluating graph dataset quality. Methodologically, RINGS introduces dual-modal perturbations—structural and feature-level—to quantify dataset discriminative power via two orthogonal metrics: *performance separability* and *modality complementarity*. Our contributions are threefold: (1) we establish the first systematic, dataset-centric evaluation paradigm—distinct from model-centric benchmarks; (2) we design novel diagnostic tools, including modality ablation, dual-modal sensitivity analysis, benchmark robustness diagnosis, and controlled degradation experiments; and (3) empirical evaluation across 12 mainstream graph datasets reveals pervasive structural redundancy and feature-dominant bias in several widely used benchmarks. Based on these findings, we provide actionable recommendations for dataset curation, design, and quality assurance.

Communication barriers between data scientists and domain experts arise from oversimplified, accuracy-centric model performance reporting, hindering shared understanding of model limitations and contextual applicability. Method: We propose a visualization-mediated model explanation framework grounded in human-computer interaction principles, participatory design, and visual narrative techniques. This yields the first domain-expert-oriented model communication guideline—emphasizing risk, trade-offs, and situational appropriateness rather than isolated metrics like accuracy. An iterative empirical study was conducted using regression models, incorporating structured expert feedback for evaluation. Contribution/Results: The framework significantly improves domain experts’ ability to identify model limitations, recognize inherent trade-offs, and proactively make context-driven adoption decisions. Its core innovation lies in repositioning visualization as an interdisciplinary consensus-building medium—shifting the paradigm from “metric reporting” to “collaborative understanding.”

The zero-shot generalization mechanism of knowledge graph foundation models on unseen graphs remains poorly understood, particularly under scenarios with partially observed relations (semi-links), where performance is often unstable. This work introduces four fine-grained generalization settings based on the visibility of head-relation or relation-tail semi-links, establishes a diagnostic evaluation protocol, and systematically assesses state-of-the-art models using a hierarchical zero-shot link prediction strategy. Experimental results reveal that current advanced models heavily rely on observed semi-links for inference and suffer significant performance degradation when confronted with entirely unseen semi-links. These findings underscore the critical role of semi-link visibility in zero-shot generalization and provide clear guidance for future model design.

This study addresses the lack of a unified and general evaluation framework for explainability in graph neural networks (GNNs), which hinders cross-model comparisons. To bridge this gap, the authors propose the AIM evaluation framework, which systematically assesses GNN explainability along three dimensions: accuracy, instance-level fidelity, and model-level interpretability, thereby establishing the first general-purpose benchmark applicable across diverse GNN architectures. Applying AIM to intrinsically interpretable models such as Graph Kernel Networks (GKNs) reveals inherent limitations in their explanations. Motivated by these insights, the authors design an enhanced model, xGKN, which significantly improves explainability while maintaining high predictive performance. This work not only introduces a novel paradigm for evaluating GNN explainability but also demonstrates how principled evaluation can directly inform and drive model improvement.

Current AI system evaluations suffer from fragmented assessment dimensions, heterogeneous evidence sources, and insufficient transparency. To address these challenges, this paper proposes the “Measurement Tree”—a novel multi-source fusion evaluation framework based on a hierarchical directed graph. Structured as a tree-like data model, it supports user-defined aggregation functions to unify heterogeneous metrics—including agency, business value, energy efficiency, socio-technical impact, and safety—into interpretable, multi-level representations. This work introduces, for the first time, a hierarchical graph structure as the formal output format for AI evaluation, substantially enhancing traceability and interpretability. An accompanying open-source Python library and extensive empirical validation demonstrate that the Measurement Tree improves comprehensiveness, operationality, and reproducibility in evaluating complex AI systems. It thus provides foundational infrastructure for building an open and transparent AI evaluation ecosystem.

Current machine learning evaluation practices predominantly rely on surface-level performance metrics, often neglecting the internal mechanisms of models. This work proposes trustworthy interpretability as a central evaluation paradigm and, for the first time, systematically demonstrates that it satisfies core criteria from the philosophy of science—namely falsifiability, reproducibility, and predictive power. By constructing an evaluation framework that integrates causal analysis with mechanistic probing, the study delineates three functional pathways through which interpretability enables the identification of behavioral origins, detection of latent flaws, and prediction of potential failure modes. This approach advances model assessment beyond performance-oriented benchmarks toward a deeper understanding of underlying mechanisms.

Existing LLM-KG evaluation benchmarks overemphasize answer accuracy while neglecting systematic characterization of task-level cognitive complexity, leading to fragmented capability assessment, undetected blind spots, and insufficient task diversity. Method: This work introduces, for the first time, three established cognitive complexity frameworks from cognitive psychology into LLM-KG benchmark analysis, enabling multidimensional complexity modeling of knowledge graph (KG) tasks on LLM-KG-Bench. Contribution/Results: We uncover severe imbalances in cognitive demand distribution across current evaluations—particularly underrepresentation of higher-order reasoning and multi-step planning tasks. Our findings provide empirical grounding and a principled design paradigm for developing more interpretable, balanced, and challenging KG evaluation tasks. This advances KG evaluation from a purely outcome-oriented paradigm toward a dual-dimensional framework that jointly assesses both process-level reasoning and underlying cognitive capabilities.

Existing Accumulated Local Effects (ALE) methods cannot be directly applied to link prediction in Graph Neural Networks (GNNs) due to nonlinear, high-order feature interactions among nodes induced by message passing. Method: We propose a scalable approximate ALE computation framework for GNNs, leveraging local subgraph sampling and gradient proxy strategies to preserve the causal semantics of ALE while drastically reducing computational complexity. Contribution/Results: Experiments on GCN and GAT demonstrate that the approximate ALE achieves explanation fidelity statistically indistinguishable from exact ALE (p > 0.05), with inference speedups of 1–2 orders of magnitude. Exact ALE exhibits greater robustness under limited samples, revealing an inherent trade-off between interpretability and efficiency. To our knowledge, this is the first work to systematically adapt ALE to GNN-based link prediction, providing an efficient, reliable, and model-agnostic visualization tool for local attribution in graph models.