Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

2026-08-03

AI Summary

This work addresses the low accuracy—under 50%—of large language models (LLMs) in filling tool parameters within complex domains such as cloud networking. The authors propose a unified framework that, for the first time, reveals strong correctness signals embedded in LLM hidden states. Leveraging this insight, they construct a linear probe and integrate it into a novel pipeline featuring probe-guided bootstrapped training (PBT) and probe-guided re-ranking (PGR) during inference. Additionally, they introduce ParamBench, the first benchmark annotated by parameter nesting depth, inter-parameter dependencies, and reasoning complexity. Evaluated on ParamBench and six external benchmarks, the proposed method substantially improves the average exact match accuracy of five open-source LLMs from 19.7% to 59.6%.

Abstract

Large language model agents derive much of their capability from tool use. Existing research on tool use has largely focused on selecting the right tool and orchestrating the order of calls. However, correctly filling the parameters of a tool call is equally critical for successful execution and has received far less attention. In domains such as cloud networking, even frontier models correctly complete fewer than half of tool calls. Inspired by recent analyses showing that LLM hidden states encode rich information about model predictions, we discover that while the model generates a parameter value, its hidden state contains a strong correctness signal: a simple linear probe can accurately predict whether the value will be correct. Based on this observation, we propose a unified probe-guided framework with two complementary approaches: probe-filtered bootstrapped training (PBT), which uses the probe to filter reliable self-generated calls for fine-tuning, and probe-guided reranking (PGR), which uses the probe to select better candidates during inference. To support systematic evaluation, we release ParamBench, a benchmark built from real cloud-network APIs that categorizes every instance into five difficulty levels according to parameter nesting depth, cross-parameter dependencies, and the reasoning required to derive values from earlier calls. Extensive experiments across 5 open models on ParamBench and 6 external benchmarks demonstrate that our method substantially improves parameter generation, raising the average exact match from 19.7% to 59.6%.

Problem

• tool use
• parameter filling
• LLM agents
• correctness prediction
• difficulty-graded benchmark

Innovation

• probe-guided training
• tool call parameters
• hidden state probing
• ParamBench
• LLM tool use