🤖 AI Summary
This work addresses a critical gap in the evaluation of large language models (LLMs) as tool users by shifting focus from mere task success rates to the underlying cognitive mechanisms of tool discovery. Inspired by cognitive science, it proposes the first systematic framework that decomposes tool discovery into three quantifiable dimensions: curiosity, recognition capability, and usage efficiency. Through empirical analysis in both combinatorial game settings and realistic simulated environments such as Voyager, the study uncovers a counterintuitive inverse scaling law: larger models exhibit diminished recognition capability despite their increased scale. The proposed framework not only offers a more fine-grained lens for evaluating LLMs’ tool-use proficiency but also demonstrates robust validity and generalizability across diverse experimental settings.
📝 Abstract
Existing tool-use benchmarks report a single success rate for complex, multistep tasks. Inspired by ideas from cognitive science, we distinguish tool use from tool discovery and decompose the latter into curiosity (the model's ability to discover the parts needed to build the tool), recognition (the model's ability to discover the process of creating the tool), and efficiency (the model's use of the tool after creation). We show that this framework can be applied to existing discovery tasks, such as Voyager. In addition, we provide evidence that recognition inversely scales with model size, and we introduce and analyze a class of combinatorial games that demonstrates this. We further observe inverse scaling in a separate environment designed to emulate real-world tasks.