Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind

2026-02-14

AI Summary

This work addresses the challenge that large language models struggle to accurately infer users’ true intentions when their expressions are ambiguous, leading to cognitive discrepancies between users’ subjective beliefs and the actual environmental state. To bridge this gap, the paper extends Theory of Mind (ToM) from isolated belief reasoning to an interactive, pragmatic mechanism by proposing a novel framework for detecting and resolving cognitive discrepancies in real-world tasks. Key contributions include the construction of the first trajectory-level ToM dataset, the design of a benchmark for evaluating cognitive discrepancy awareness, and the integration of reinforcement learning to enhance models’ reasoning about users’ mental states. Experiments across 11 mainstream large language models reveal a widespread deficiency in recognizing such cognitive gaps, while models trained on the proposed dataset demonstrate significantly improved mental-state reasoning and downstream task performance.

Technology Category

Cognitive Modeling & Cognitive Systems: Conceptual Inference and ReasoningNatural Language Processing: (Large) Language ModelsKnowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models (LLMs) have developed rapidly and are widely applied to both general-purpose and professional tasks to assist human users. However, they still struggle to comprehend and respond to the true user needs when intentions and instructions are imprecisely conveyed, leading to a divergence between subjective user believes and true environment states. Resolving this epistemic divergence requires Theory of Mind (ToM), yet existing ToM evaluations for LLMs primarily focus on isolated belief inference, overlooking its functional utility in real-world interaction. To this end, we formalize ToM for LLMs as a mechanism for epistemic divergence detection and resolution, and propose a benchmark, \benchname, to assess how models reconcile user beliefs and profiles in practice. Results across 11 leading models reveal a significant limitation to identify underlying cognitive gaps that impede task success. To bridge this gap, we further curate a trajectory-based ToM dataset linking belief tracking with task-related state inference. The model trained on this data via reinforcement learning shows consistent improvement in reasoning about user mental states, leading to enhanced downstream performance. Our work highlights the practical value of ToM as an essential interaction-level mechanism rather than as a standalone reasoning skill.

Problem

• epistemic divergence
• Theory of Mind
• user-agent interaction
• belief inference
• cognitive gaps

Innovation

• Theory of Mind
• Epistemic Divergence
• User-Agent Interaction
• Trajectory-based Dataset
• Reinforcement Learning