HALT: Hallucination Assessment via Log-probs as Time series

2026-02-02

AI Summary

This work addresses the critical challenge of hallucination in large language models (LLMs) within safety-sensitive applications by proposing a lightweight, general-purpose detection framework. The method constructs a time series from the log-probabilities of only the first 20 generated tokens at the output layer and employs a gated recurrent unit (GRU) combined with entropy-based features to learn and calibrate bias, without requiring access to internal model states or surface-level textual features. Evaluated on the unified HUB benchmark, the approach achieves superior performance: its model is 30 times smaller and 60 times faster at inference than Lettuce, while demonstrating stronger generalization across ten diverse language tasks, making it particularly suitable for closed-source LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Hallucinations remain a major obstacle for large language models (LLMs), especially in safety-critical domains. We present HALT (Hallucination Assessment via Log-probs as Time series), a lightweight hallucination detector that leverages only the top-20 token log-probabilities from LLM generations as a time series. HALT uses a gated recurrent unit model combined with entropy-based features to learn model calibration bias, providing an extremely efficient alternative to large encoders. Unlike white-box approaches, HALT does not require access to hidden states or attention maps, relying only on output log-probabilities. Unlike black-box approaches, it operates on log-probs rather than surface-form text, which enables stronger domain generalization and compatibility with proprietary LLMs without requiring access to internal weights. To benchmark performance, we introduce HUB (Hallucination detection Unified Benchmark), which consolidates prior datasets into ten capabilities covering both reasoning tasks (Algorithmic, Commonsense, Mathematical, Symbolic, Code Generation) and general purpose skills (Chat, Data-to-Text, Question Answering, Summarization, World Knowledge). While being 30x smaller, HALT outperforms Lettuce, a fine-tuned modernBERT-base encoder, achieving a 60x speedup gain on HUB. HALT and HUB together establish an effective framework for hallucination detection across diverse LLM capabilities.

Problem

• hallucination
• large language models
• log-probabilities
• domain generalization
• safety-critical

Innovation

• hallucination detection
• log-probability time series
• lightweight LLM monitoring
• model calibration
• domain generalization