ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

2026-06-09

AI Summary

This work addresses the lack of structured user intent, multi-turn task delegation, and authentic tool execution in existing OS agent training data. To bridge this gap, we propose ISE, a three-stage synthetic data generation paradigm: first, we construct a structured intent pool grounded in a 4D intent framework (Persona × Domain × Task × Complexity); second, we employ a role-locked user simulator to generate multi-turn interaction trajectories; and third, we execute all tool calls within a real, isolated OS sandbox to ensure data authenticity and diversity. ISE is the first end-to-end framework integrating structured intent generation, execution-driven dialogue simulation, and real tool invocation, substantially enhancing agents’ tool-use proficiency. Fine-tuning Qwen3-8B on ISETrace achieves a 37.7% pass@1 on ClawEval, outperforming both zero-shot GPT-4o and the base Qwen3-32B model with four times the parameters.

Abstract

Training capable OS agents requires data that simultaneously captures structured user intents, multi-turn task delegation, and grounded tool execution--properties absent from existing datasets. We propose ISE (Intent -> Simulate -> Execute), a three-stage synthesis paradigm that addresses these gaps jointly. Stage 1 constructs roughly 50000 structured intents via a 4D framework (Persona x Domain x Task x Complexity); after deduplication the pool contains 43956 unique intents and attains a Vendi Score of 61.57 over the entire pool on mpnet-base-v2 embeddings (cosine kernel, q=1). Stage 2 drives multi-turn user-agent interaction through a role-locked user simulator that grounds each user turn in actual execution outcomes, producing 23132 complete trajectories averaging 8.12 user turns and 68.24 total dialogue turns. Stage 3 runs every tool call inside a live, isolated OS workspace, generating authentic failure-recovery dynamics instead of simulated responses. Fine-tuning on ISETrace improves ClawEval pass@1 from 19.3 to 37.7 using Qwen3-8B on agent tool-use tasks with a standard protocol. This result outperforms zero-shot GPT-4o and the larger Qwen3-32B base model which is four times bigger. An ablation on Stage 2 proves multi-turn simulation brings a large portion of the performance gain. We release all source code and dataset at https://github.com/Valiere01/ISE-Trace.

Problem

• OS agents
• structured user intents
• multi-turn task delegation
• grounded tool execution
• agent training data

Innovation

• multi-turn simulation
• execution-grounded trajectories
• structured intent synthesis
• OS agent training
• tool-use failure recovery