Framework for Machine Evaluation of Reasoning Completeness in Large Language Models For Classification Tasks

2025-10-23

AI Summary

This work investigates whether explanations generated by large language models (LLMs) for text classification tasks faithfully reflect their actual decision-making rationale, and evaluates their consistency with interpretable baselines such as logistic regression. To this end, we propose RACE—a novel framework that quantifies, for the first time, the alignment between LLM reasoning and ground-truth salient features via multi-granularity matching techniques, including token-level alignment, exact string matching, and edit-distance-based similarity. Experiments across four benchmark text classification datasets reveal that while LLMs effectively reuse supportive evidence under correct predictions, they disproportionately amplify contradictory features under incorrect ones, leading to substantial explanation infidelity. Moreover, we uncover an asymmetry in evidence coverage: supportive features are more consistently identified in correct predictions than in errors. Our study establishes a new empirical paradigm for evaluating LLM interpretability and provides foundational insights into the fidelity of model-generated explanations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The growing adoption of machine learning (ML) in sensitive domains has heightened the demand for transparent and interpretable artificial intelligence. Large Language Models (LLMs) are increasingly capable of producing natural language explanations, yet it remains unclear whether these rationales faithfully capture the predictive signals that underlie decisions. This paper introduces RACE-Reasoning Alignment for Completeness of Explanations, a systematic framework to evaluate the alignment between LLM-generated explanations and interpretable feature importance scores derived from a logistic regression baseline. We analyze four widely used text classification datasets-WIKI ONTOLOGY, AG NEWS, IMDB, and GOEMOTIONS-and compare LLM rationales against top-ranked supporting and contradicting lexical features. To capture alignment at multiple levels of granularity, RACE implements token-aware, exact string, and edit-distance matching techniques. Empirical results reveal a consistent asymmetry: correct predictions exhibit higher coverage of supporting features, while incorrect predictions are associated with elevated coverage of contradicting features. Edit-distance matching further uncovers paraphrastic overlaps, boosting coverage while preserving this asymmetry. These findings demonstrate that LLM rationales combine both surface-level and flexible evidence reuse, yet can also amplify misleading cues in error cases. RACE provides new insights into the faithfulness of LLM explanations and establishes a quantitative basis for evaluating reasoning completeness in neural language models.

Problem

• Evaluating alignment between LLM explanations and interpretable feature importance scores
• Assessing reasoning completeness in neural language models for classification tasks
• Analyzing faithful capture of predictive signals in LLM-generated rationales

Innovation

• Framework evaluates explanation-feature alignment in LLMs
• Uses token-aware and edit-distance matching techniques
• Quantitatively assesses reasoning completeness via feature coverage