VC-Tooler: Learning Compositional and Adaptive Visual Tool Use

2026-08-03

AI Summary

Existing vision-language models struggle to effectively compose multiple visual tools and dynamically adjust their reasoning based on feedback. This work presents the first systematic approach to modeling the compositionality and adaptability of visual tool usage, introducing a hierarchical synthetic trajectory construction method and a two-stage training paradigm comprising supervised pretraining followed by reinforcement learning. Central to the framework is a multi-tool composition scheduling mechanism that enables flexible and context-aware tool integration. The proposed method achieves state-of-the-art performance among open-source models, attaining 95.8% accuracy on the V* benchmark and 35.3% on VTC-Bench, while demonstrating strong generalization capabilities in more complex tool-rich environments.

Abstract

Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including $95.8\%$ on V* and $35.3\%$ on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler

Problem

• visual tool use
• compositional reasoning
• adaptive reasoning
• multimodal agents
• tool interaction

Innovation

• compositional reasoning
• adaptive tool use
• visual tool interaction
• trajectory synthesis
• reinforcement learning