Problem
Research questions and friction points this paper is trying to address.
Next-Turn Metrics
Workflow Success
Autonomous Execution
When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
🤖 AI Summary
研究探讨了提高单次决策准确性是否能提升自主工作流执行成功率，通过对比分析不同模型在客户支持多轮对话中的表现，发现单次决策改进并不保证整体流程成功。
📝 Abstract
Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is scored against a reference. We investigate whether improvement under this protocol is predictive of improved autonomous workflow execution. We study pre-SFT and supervised fine-tuned (SFT) Qwen3 models at 4B and 14B parameters and Gemma 3 models at 4B and 12B parameters on multi-turn customer-support workflows. We find that SFT consistently improves text-turn success, and that overall next-turn success increases for every model under gold-history evaluation. However, these improvements do not transfer to autonomous workflow execution. Tool-specific gains also vary across metrics and models. None of the four SFT models succeeds under holistic workflow evaluation, with strict trajectory completion reaching at most 10.4% workflow success. Our results show that next-turn evaluation is not a reliable proxy for workflow success, motivating separate reporting of text quality, local action correctness, tool execution, and end-to-end task completion.
Innovation
Methods, ideas, or system contributions that make the work stand out.
next-turn evaluation
workflow success
supervised fine-tuning (SFT)
autonomous workflow execution
end-to-end task completion
🔎 Similar Papers
No similar papers found.
