OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

2026-06-05

AI Summary

Existing hallucination detection evaluations suffer from inconsistent inference setups and evaluation criteria, as well as insufficient coverage of downstream tasks, leading to challenges in performance comparability, reproducibility, and generalization. To address these limitations, this work proposes OpenHalDet—the first unified benchmark for hallucination detection that supports cross-task, cross-model, and cross-paradigm (black-box, gray-box, and white-box) evaluation. The framework standardizes the entire pipeline, from prompt construction and response generation to factuality annotation, detection scoring, and metric computation. Furthermore, it provides an open-source, extensible platform enabling fair and systematic assessment of heterogeneous detection methods across diverse generative scenarios, significantly enhancing the comparability, reproducibility, and generalizability of hallucination detection evaluations.

Abstract

Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challenges: inconsistent inference configuration and evaluation, and limited coverage of downstream domains and tasks. Consequently, reported detector performance is often difficult to compare, reproduce, and generalize beyond specific experimental settings. We introduce OpenHalDet, a unified benchmark for hallucination detection across diverse generation scenarios. OpenHalDet standardizes the evaluation pipeline, from prompt construction and response generation to truthfulness annotation, detector scoring, and metric computation. It supports heterogeneous detector families under different access settings, including black-box methods that use only generated outputs, gray-box methods that rely on probability-based signals, and white-box methods that exploit internal model signals. By bringing diverse tasks, models, and detectors into a shared framework, OpenHalDet enables controlled comparison and provides a systematic view of how different detection paradigms behave in LLM applications. We release OpenHalDet as an open and extensible codebase to facilitate reproducible evaluation and future development of hallucination detection methods. The code and datasets are available at https://github.com/Nellie179/Hallucination-Detection.

Problem

• hallucination detection
• large language models
• evaluation benchmark
• reproducibility
• generalization

Innovation

• hallucination detection
• unified benchmark
• large language models
• evaluation standardization
• detector paradigms