XAI Evaluation Cards: A Practical Method for Designing Human-Centred XAI Evaluations

2026-10-01

AI Summary

This study addresses the fragmented dimension selection, lack of systematicity, and insufficient human-centered perspectives in Explainable Artificial Intelligence (XAI) evaluation by constructing a systematic assessment framework grounded in a bibliometric analysis of 82 studies. The core innovation lies in developing a card-sorting method comprising 36 cards that concretizes abstract evaluation dimensions, thereby assisting researchers in efficiently designing human-centered XAI evaluation procedures. Empirically validated through two groups of researchers, this approach effectively streamlines the evaluation design process, facilitates multidisciplinary collaboration, and significantly enhances both the comprehensiveness and structural rigor of XAI system assessments.

Abstract

Evaluating explainable AI (XAI) systems from a human-centred approach requires researchers to select from numerous evaluation dimensions and measures, often in an ad hoc and fragmented manner. This paper introduces a method to help HCI, computer science, designers and social science researchers systematically evaluate XAI systems. The approach is based on an updated XAI-specific evaluation framework derived from an analysis of 82 studies. Using this framework, we developed a card-sorting method with 36 cards to help researchers prioritise relevant evaluation aspects. The process was tested with two research groups (n = 13) across five projects. The XAI Evaluation Cards are available as a printable appendix, along with an online repository of methods from previous XAI studies. Although not exhaustive, our findings indicate that the card-sorting approach can organise and streamline the design of the evaluation process, encouraging a more comprehensive and multidisciplinary assessment of XAI systems in research and development.

Problem

• Explainable AI
• Human-centred evaluation
• Evaluation dimensions
• XAI assessment

Innovation

• Explainable AI (XAI)
• Human-Centred Evaluation
• Card-Sorting Method
• Evaluation Framework
• Multidisciplinary Assessment