HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

2026-05-04

AI Summary

This study addresses the susceptibility of large language models to hallucinations—generating content inconsistent with facts, context, or user intent—by introducing HalluScan, a benchmark framework for systematically evaluating hallucination detection and mitigation across 72 configurations spanning six detection methods, four open-source model families, and three domains. The work proposes HalluScore, a composite metric highly correlated with expert judgments (r = 0.41), an adaptive detection routing algorithm (ADR) that halves inference cost with only a 0.1% drop in AUROC, and a cross-domain decomposition of hallucination error types. Experiments demonstrate that natural language inference–based NLI Verification achieves state-of-the-art performance (AUROC = 0.88), substantially outperforming RAV (AUROC = 0.66), thereby validating the framework’s advantages in both accuracy and computational efficiency.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.

Problem

• hallucination
• instruction-following LLMs
• factuality
• benchmark
• error detection

Innovation

• HalluScore
• Adaptive Detection Routing
• hallucination benchmarking
• error cascade decomposition
• instruction-following LLMs