CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

2026-09-27

AI Summary

This study addresses the limitation of existing metrics in summary faithfulness evaluation, which focus solely on isolated hallucinations while neglecting interactions among entity, relation, and numerical errors. To this end, we propose the Comprehensive Hallucination Index (CHI), a method that decomposes errors into three statistically orthogonal dimensions. CHI employs a Venn-diagram-factor-based Softmax normalization architecture alongside tolerance-aware numerical matching, integrating these components via harmonic mean to generate a unified score. Experimental results demonstrate that CHI achieves a system-level correlation of 0.66 on the SummEval dataset, significantly outperforming baseline methods such as ROUGE. Ultimately, this work enables efficient, interpretable, and diagnostic faithfulness evaluation for text summarization.

Abstract

Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated hallucination types: factual entity errors, relational inconsistencies, or numerical fabrications, without capturing their co-occurrence or interaction. We introduce CHI (Composite Hallucination Index), the first unified hallucination metric that decomposes faithfulness errors into three orthogonal dimensions: entity hallucination (EHI), relation hallucination (RHI*), and quantity hallucination (QHI). Each dimension employs a shared softmax-normalized architecture over Venn diagram-derived factors representing extractiveness, positive hallucination, over-focus, negative hallucination, and lost focus. The novel QHI component introduces tolerance-aware numerical matching with exact, epsilon, derived, and temporal comparison modes. We fuse the three dimensions via harmonic mean to produce a single composite score that penalizes weakness in any dimension. We validate CHI on 800 source articles spanning four domains (news, medical, legal, financial) with summaries from five generation systems. Empirical results demonstrate that: (i) the three dimensions are statistically orthogonal (mean rho = 0.148), confirming they capture distinct error types; (ii) CHI achieves the highest system-level correlation with human judgments (rho = 0.66, p = 0.006) on SummEval, outperforming ROUGE (rho = 0.53), EHI (rho = 0.58), and all individual components; and (iii) ablation studies confirm that all three dimensions contribute unique variance, with the full composite outperforming any individual component while providing decomposable error diagnostics unavailable from single-score baselines. CHI provides practitioners with a decomposable, interpretable, and efficient faithfulness metric suitable for both offline evaluation and online monitoring of summarization systems.

Problem

• hallucination evaluation
• faithfulness
• abstractive summarization
• composite metric

Innovation

• Composite Hallucination Index
• Faithfulness Evaluation
• Quantity Hallucination
• Tolerance-aware Matching
• Summarization