TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs

2026-08-06

AI Summary

This work addresses the challenge of object hallucination in large vision-language models (LVLMs), which undermines their reliability in real-world deployment. The authors propose TruthLens, a novel framework that enables self-assessment of hallucinations by leveraging internal signals from the language modeling head. Specifically, it repurposes rare special tokens as references and constructs a veracity score based on the log-probability differences between these tokens and standard outputs—requiring no additional models or inference overhead. With only lightweight fine-tuning guided by an MSE objective and distributional constraints, TruthLens preserves generative capabilities while generalizing effectively to broader category spaces. Evaluated across multiple LVLMs, the method achieves state-of-the-art performance, surpassing the previous best approach by over 17% in AUROC on MS-COCO using Qwen2.5-VL-7B.

Abstract

Despite the remarkable progress of large vision language models (LVLMs), object hallucination remains a fundamental challenge that hinders their trustworthy deployment. A key finding motivates our work: real and hallucinated object tokens are clearly separable in hidden representations, yet this separability is largely lost at the language-modeling (LM) head. We propose TruthLens, a self-evaluation framework that teaches the LM head to expose a per-object truthfulness signal without any auxiliary model or additional inference cost. Concretely, a rarely-used special token is repurposed as a reference token. For each object-token position, we extract the log-probability assigned to this special token by the LM head, and define its difference from a predefined constant as the truthfulness score. The model is then fine-tuned with an MSE objective that drives scores toward 1 for real objects and 0 for hallucinated ones, while a divergence constraint preserves the original generation capability. Despite being trained on only a limited set of object categories, TruthLens generalizes effectively to benchmarks with substantially larger label spaces. Extensive experiments across multiple LVLMs demonstrate state-of-the-art performance; notably, on Qwen2.5-VL-7B, TruthLens outperforms the previous best method on MS-COCO by over 17\% in AUROC. Our code is available at https://github.com/wyqstan/TruthLens.

Problem

• object hallucination
• large vision language models
• truthfulness
• hallucination detection
• trustworthy AI

Innovation

• object hallucination detection
• self-evaluating truthfulness
• large vision language models
• reference token
• truthfulness score