Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

2026-07-29

AI Summary

This work addresses the challenge of balancing informational gains against heterogeneous costs—such as computational overhead and privacy risks—when large language model agents invoke external tools, a setting where existing approaches lack optimal stopping mechanisms. The authors propose the Cost-Aware Marginal Decision-Focused stopping framework (CAM-DF) and its lightweight, interpretable variant CAM-DF-lite, which formulate tool acquisition as an offline optimization problem that directly minimizes the gap between a stopping decision and the optimal continuation policy. Requiring no fine-tuning of the underlying language model, CAM-DF relies solely on symbolic labels and error-weighted training, enabling it to serve as a plug-and-play module compatible with any tool ranker. Experiments across 1,343 tasks demonstrate that CAM-DF significantly outperforms baselines under high-cost or weak-ranking conditions, achieving the highest utility gain on τ-bench Retail while reducing tool calls by 37% without compromising task success rates.

Abstract

As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure. Routers and retrievers can rank candidate tools by relevance, but a ranking alone does not determine how many are worth selecting. Existing approaches leave acquisition under heterogeneous costs unaddressed. We formulate this decision as cost-aware marginal decision-focused stopping (CAM-DF) over ranked tool prefixes, with CAM-DF-lite as a compact interpretable variant. We train directly on the offline gap between stopping now and the best continuation: its sign labels the decision, its magnitude weights each error by the payoff at stake. We prove this objective is Bayes-aligned with the stopping target and that score-only rules are suboptimal under heterogeneous costs. We evaluate on 1,343 tasks across five tool-use domains. On $τ$-bench Retail, CAM-DF attains the highest payoff among deployable methods, with gains over a predict-then-threshold baseline across all five ranking sources and two cost regimes. Our approach is state-of-the-art under heterogeneous costs and high cost pressure, with larger gains under weaker rankings. In live execution, CAM-DF exposes the agent to 37\% fewer tools than full access while maintaining comparable task success. The CAM-DF family is a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.

Problem

• tool acquisition
• cost-aware stopping
• LLM agents
• heterogeneous costs
• decision-focused stopping

Innovation

• cost-aware stopping
• tool acquisition
• LLM agents
• marginal decision-focused learning
• heterogeneous costs