Structured but Silent: Probing Capability Requirements in LLM Hidden States

2026-10-06

AI Summary

This study addresses the frequent misalignment between implicit user intent understanding and explicit expression during LLM tool calling, investigating whether models can decode query-specific capability requirements from hidden states prior to generation. To this end, it proposes the TACIT framework, which defines eight capability categories along three axes—source, transformation, and world effects—and employs linear probing to conduct fine-grained decoding and comparative analysis of pre-generation hidden states across multiple open-source LLM families. The work reveals a phenomenon termed “structured silence,” demonstrating that while models internally encode structured capability information that is linearly decodable with high precision, they cannot reliably verbalize such classifications explicitly. This finding substantiates a significant gap between internal representations and verbalization, offering new insights into the mechanisms underlying implicit knowledge representation in large language models.

Abstract

Reliable tool use requires more than triggering a mechanism or matching a query to an API description. Before selecting a specific tool, an agent must first infer the capability requirements implied by the user query. In this paper, we investigate whether these query-side capability requirements are linearly decodable from LLM hidden representations prior to generation, and how this hidden-state accessibility compares with explicit verbal classification. We introduce TACIT, a framework that decomposes external requirements along three fundamental axes: Source, Transformation, and World Effect, defining eight structurally distinct capability classes. Using 1,600 balanced training queries from benchmarks, synthetic examples, and new domain scenarios, we train linear probes on pre-generation hidden states from four open-weight LLM families. Our empirical results demonstrate that fine-grained capability structures are linearly decodable with high accuracy across all models. Crucially, however, we expose a representation-to-verbalization gap: these same models are significantly less reliable when asked to explicitly classify the same queries in natural language. This disconnect indicates that information about required external capabilities is linearly accessible in LLM hidden representations but not reliably expressed, a phenomenon we define as "structured but silent."

Problem

• capability requirements
• hidden states
• tool use
• linear decodability
• representation-to-verbalization gap

Innovation

• TACIT framework
• linear probing
• hidden representations
• representation-to-verbalization gap
• structured but silent