MIMESIS: Learning User Simulators as Training Environments for Interactive Agents

2026-10-07

AI Summary

This study addresses the prohibitive cost of authentic human feedback and the limited behavioral diversity of existing simulated users in interactive agent training. To overcome these challenges, this work proposes a human dialogue- and reasoning-supervised user simulator encompassing 13 authentic user behavior patterns, alongside a Coach-guided Self-Distillation (CSD) method. CSD leverages private reasoning trajectories to generate instructional guidance notes, integrating large language model fine-tuning with multi-turn reinforcement learning to train highly generalizable agents. Experimental results demonstrate that the proposed approach achieves a SOUL-Index of 65.7, surpassing frontier models while improving behavioral fidelity by 13.4 points. Furthermore, the method exhibits significantly superior performance compared to GPT-5.5 baselines across multiple unseen scenarios, highlighting its strong generalization capabilities for interactive agent development.

Abstract

Training and evaluating interactive language agents typically requires rich user interactions, yet collecting human feedback is expensive and difficult to scale. Simulated users offer a scalable alternative, but they must both resemble real user behavior and provide useful learning experiences for agents. In contrast, most agent-training frameworks rely on off-the-shelf assistant LLMs, whose helpfulness can make them overly cooperative, explicit, and behaviorally homogeneous compared with real users. We introduce MIMESIS, a purpose-built user simulator trained on human conversations with explicit reasoning supervision and 13 realistic behavioral patterns derived from real user interactions. Empirically, our 9B model achieves a SOUL-Index of 65.7, surpassing the strongest frontier model. Compared with Claude-Opus-5, the strongest baseline on RealUserSim and SimulatorArena, MIMESIS improves behavioral fidelity by 13.4 points and reduces Turing distance by 3.6 points, respectively. We then freeze the simulator and train an agent by interacting with the frozen simulator using multi-turn reinforcement learning. Across eight environments, training with MIMESIS yields better agent performance than training with GPT-5.5 under all nine unseen user simulators, demonstrating stronger generalization to new user simulators. Moreover, we propose Coached On-Policy Self-Distillation (CSD), which leverages simulator-generated private reasoning traces and subsequent utterances as feedback on how well the agent addresses user needs. A coach converts this information into concise coaching notes that describe how the agent can better anticipate user needs and adapt its behavior over the course of an interaction. CSD turns this feedback into dense, token-level supervision beyond sparse task rewards, yielding further gains across all nine evaluation user models.

Problem

• interactive language agents
• user simulators
• human feedback
• behavioral fidelity
• simulated users

Innovation

• User Simulator
• Multi-turn Reinforcement Learning
• Self-Distillation
• Interactive Agents
• Behavioral Fidelity