🤖 AI Summary
This study addresses the challenge that post-hoc interpretability tools for credit risk models—such as SHAP—produce numerical outputs that are difficult for non-technical stakeholders to comprehend. It presents the first systematic evaluation of large language models (LLMs) in two distinct roles: as “translators” that convert attribution-based explanations into natural language narratives, and as “autonomous explainers” that generate explanations directly from input data. Using LendingClub data with logistic regression and XGBoost models, the authors experiment with few-shot prompting on GPT-4-turbo, Claude Sonnet 4, and Gemini-2.0-Flash. Results show that LLMs excel as translators, producing intelligible and auditable narrative explanations, but perform poorly as autonomous explainers, exhibiting low fidelity to model attributions—especially for nonlinear models. The work thus advocates integrating LLMs as complementary narrative interfaces to, rather than replacements for, established interpretability methods.
📝 Abstract
Post-hoc explainability is central to credit risk model governance, yet widely used tools such as coefficient-based attributions and SHapley Additive exPlanations (SHAP) often produce numerical outputs that are difficult to communicate to non-technical stakeholders. This paper investigates whether large language models (LLMs) can serve as post-hoc explainability tools for credit risk predictions through in-context learning, focusing on two roles: translators and autonomous explainers. Using a personal lending dataset from LendingClub, we evaluate three commercial LLMs, including GPT-4-turbo, Claude Sonnet 4, and Gemini-2.0-Flash. Results provide strong evidence for the translator role. In contrast, autonomous explanations show low alignment with model-based attributions. Few-shot prompting improves feature overlap for logistic regression but does not consistently benefit XGBoost, suggesting that LLMs have limited capacity to recover non-linear, interaction-driven reasoning from prompt cues alone. Our findings position LLMs as effective narrative interfaces grounded in auditable model attributions, rather than as substitutes for post-hoc explainers in credit risk model governance. Practitioners should leverage LLMs to bridge the communication gap between complex model outputs and regulatory or business stakeholders, while preserving the rigor and traceability required by credit risk governance frameworks.