🤖 AI Summary
This work addresses the challenges faced by multi-turn tool-using agents in long-horizon tasks, where coordinating tool sequences, tracking states, and enforcing strategic constraints are difficult to unify. Existing approaches suffer from a disconnect between reasoning and learning, leading to suboptimal tool selection and preference learning vulnerable to prompt misalignment. To overcome these limitations, the authors propose ToolGraph, a novel framework that integrates tool graph topology with divergence-point localization. By leveraging state matching and prefix alignment, ToolGraph identifies trajectory divergences and employs action correctness filtering to construct high-quality preference pairs, enabling context-consistent Direct Preference Optimization (DPO). Evaluated on 375 tau2-bench tasks, ToolGraph improves the weighted average reward from 0.304 to 0.338 (+11.2%), and further to 0.355 (+16.8%) when combined with DPO, substantially outperforming baselines—particularly in aviation and retail scenarios.
📝 Abstract
Multi-turn tool-using agents must coordinate long-horizon tool sequences while tracking dialogue state and policy constraints. Existing approaches often separate inference-time orchestration from parameter-level learning, leaving tool selection weakly structured and preference updates vulnerable to train--deployment prompt mismatch. For within-benchmark self-improvement, ToolGraph combines schema-derived topology, transition weights estimated from successful rollouts, and history-aware controls for write prerequisites and repeated-search loops. We then construct 161 preference pairs by locating divergence points via state-based matching and prefix-based alignment, filtered through action-correctness annotations, and train DPO under the same ToolGraph context used at inference. Across 375 tau2-bench tasks, ToolGraph raises the weighted average reward from 0.304 to 0.338 (+11.2% relative), while ToolGraph+DPO reaches 0.355 (+16.8% over the baseline), with the DPO gain concentrated in airline and retail. Fine-grained diagnostics further show that roughly half of telecom trajectories exhaust the step budget before action execution and that chosen reward positivity is the most useful checkpoint signal across our 16 evaluated DPO configurations.