LLMs Get Lost in Evolving User Intent

2026-07-22

AI Summary

Large language models excel in static, single-turn tasks but struggle to effectively track and adapt to the evolving nature of user intent in dynamic, multi-turn dialogues. To address this limitation, this work proposes an evaluation framework that reformulates static tasks as dynamic multi-turn conversations, incorporating an intent evolution simulation mechanism that gradually reveals, revises, or shifts user intent throughout the interaction while remaining compatible with existing evaluation protocols. This framework systematically uncovers a performance gap in current models under dynamic intent scenarios, demonstrating significant discrepancies between static and dynamic settings across multiple tasks. The findings highlight a critical shortcoming in models’ ability to support collaborative interaction and establish a reusable paradigm for dynamic evaluation in future research.

Abstract

As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns--incrementally revealed, revised, and at times redirected mid-conversation--while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents.

Problem

• evolving user intent
• large language models
• dynamic interaction
• multi-turn conversation
• intent tracking

Innovation

• evolving user intent
• dynamic multi-turn evaluation
• LLM collaboration
• intent tracking
• benchmark adaptation