RealityTest: How People Probe AI Identity and Whether Models Disclose It

2026-05-29

AI Summary

Existing evaluations of AI identity disclosure are largely confined to English, synthetic queries, and text-only settings, failing to capture real-world identity probing behaviors in authentic conversations. This work proposes RealityTest—the first large-scale, multilingual, multimodal benchmark for assessing AI identity disclosure grounded in real human interactions—drawing on data from approximately 750 participants across 49 countries. The study systematically evaluates 17 text-based and 6 speech-based models, revealing that only 31% of users directly inquire about an AI’s identity in ambiguous contexts, and that real-world questions exhibit substantially greater diversity than synthetic ones. Notably, a single suppression instruction reduces disclosure rates below 30%, exposing significant biases in current safety assessments due to narrow evaluation data. This work provides the first empirical evidence of the critical influence of question phrasing and contextual framing on AI identity disclosure.

Abstract

AI systems are increasingly deployed in conversational settings where users may be uncertain whether they are speaking with a human or an AI. Despite mounting regulatory attention to this known safety risk, existing evaluations of AI disclosure are typically English-only, based on machine-generated questions, and restricted to text. We present RealityTest to comprehensively test whether AI systems disclose their identity when asked. The benchmark is the first large-scale multimodal and multilingual evaluation, grounded in human data on how people actually encounter and question AI identity in the real-world. Alongside the benchmark, we release the underlying dataset of 3,152 identity-probing queries collected from ~750 participants across 49 countries and five languages, in text and speech scenarios. We find that only 31% of people ask about identity directly in ambiguous scenarios, and that the questions people ask are far more diverse than machine-generated queries. We test 17 text and 6 speech models, and find substantial variation in disclosure behaviour. However, a single suppression instruction reduces disclosure rates to below 30%, even in the best-performing models. Validating our investment in diverse, human-grounded evaluation data, we find that how the question is phrased and the context of the conversation matter more for disclosure than which model is being tested. Safety evaluations built on narrow or synthetic query sets risk mischaracterising how models behave in realistic deployment settings.

Problem

• AI identity disclosure
• human-AI interaction
• multimodal evaluation
• multilingual benchmark
• conversational AI safety

Innovation

• multimodal evaluation
• multilingual benchmark
• human-grounded queries
• AI identity disclosure
• RealityTest