Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset

2025-03-31

AI Summary

Explanations generated by large language models (LLMs) are often unreliable, and users lack effective means to assess their quality. Method: We propose Rubrik, an education-inspired multidimensional evaluation rubric, and introduce CUBE—a benchmark dataset comprising 26K explanations spanning four reasoning and language tasks, generated by both human annotators and LLMs (including GPT, Llama, and Claude). CUBE features fine-grained, interpretable, and actionable annotations—covering clarity, conciseness, logical coherence, and more—produced via double-blind human annotation. Contribution/Results: Our analysis reveals redundancy as the primary cause of low explanation quality, with conciseness being the most deficient dimension; task type and perceived difficulty significantly influence explanation quality. All data, the Rubrik framework, and implementation code are publicly released, establishing a foundational benchmark and empirical basis for trustworthy explanation evaluation.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The performance and usability of Large-Language Models (LLMs) are driving their use in explanation generation tasks. However, despite their widespread adoption, LLM explanations have been found to be unreliable, making it difficult for users to distinguish good from bad explanations. To address this issue, we present Rubrik's CUBE, an education-inspired rubric and a dataset of 26k explanations, written and later quality-annotated using the rubric by both humans and six open- and closed-source LLMs. The CUBE dataset focuses on two reasoning and two language tasks, providing the necessary diversity for us to effectively test our proposed rubric. Using Rubrik, we find that explanations are influenced by both task and perceived difficulty. Low quality stems primarily from a lack of conciseness in LLM-generated explanations, rather than cohesion and word choice. The full dataset, rubric, and code will be made available upon acceptance.

Problem

• Evaluating reliability of LLM-generated explanations
• Assessing quality differences in human vs LLM explanations
• Identifying key flaws in LLM explanation conciseness

Innovation

• Introduces Rubrik's CUBE for explanation evaluation
• Uses 26k human and LLM-annotated explanations dataset
• Identifies conciseness as key quality factor