🤖 AI Summary
SHAP-based visualizations of feature importance lack semantically meaningful explanations for non-technical users, undermining interpretability and trust. To address this, we propose LLM-SHAP—a framework that tightly integrates large language models (e.g., GPT) with SHAP, dynamically injecting user-provided feature aliases, domain-specific descriptions, and contextual background into the explanation generation process. This enables context-aware, natural-language interpretations without model retraining, synergistically enhancing both visual and textual explanations. We evaluate LLM-SHAP in a medical diagnosis use case via a user study. Results demonstrate statistically significant improvements over standard SHAP visualizations: +38.2% higher perceived interpretability and +41.5% greater contextual relevance among end users. Our approach establishes a novel paradigm for deploying explainable AI in real-world, non-expert settings—bridging the gap between technical model outputs and human-understandable, actionable insights.
📝 Abstract
Explainable Artificial Intelligence (XAI) has become an increasingly important area of research, particularly as machine learning models are deployed in high-stakes domains. Among various XAI approaches, SHAP (SHapley Additive exPlanations) has gained prominence due to its ability to provide both global and local explanations across different machine learning models. While SHAP effectively visualizes feature importance, it often lacks contextual explanations that are meaningful for end-users, especially those without technical backgrounds. To address this gap, we propose a Python package that extends SHAP by integrating it with a large language model (LLM), specifically OpenAI's GPT, to generate contextualized textual explanations. This integration is guided by user-defined parameters (such as feature aliases, descriptions, and additional background) to tailor the explanation to both the model context and the user perspective. We hypothesize that this enhancement can improve the perceived understandability of SHAP explanations. To evaluate the effectiveness of the proposed package, we applied it in a healthcare-related case study and conducted user evaluations involving real end-users. The results, based on Likert-scale surveys and follow-up interviews, indicate that the generated explanations were perceived as more understandable and contextually appropriate compared to visual-only outputs. While the findings are preliminary, they suggest that combining visualization with contextualized text may support more user-friendly and trustworthy model explanations.