🤖 AI Summary
This work proposes an agent evaluation framework grounded in the principle that “compression is intelligence,” quantifying an agent’s intelligence by the number of bits required to reconstruct target objects under fixed task distributions, interfaces, and computational budgets. It systematically introduces an information-theoretic compression perspective into agent assessment for the first time, employing arithmetic coding, seed encoding, and fallback mechanisms to effectively measure how agent components—such as tool use, environmental interaction, and multi-step reasoning—contribute to reduced code length (i.e., decreased uncertainty). The approach is validated across diverse tasks including reversed text generation, chess move prediction, protein sequence modeling, retrieval-augmented question answering, and semantic story compression. Experimental results demonstrate that agents endowed with advanced capabilities consistently achieve significantly higher information efficiency.
📝 Abstract
Large language models are turning from isolated predictors into agentic systems: they call tools, retrieve evidence, obey environment constraints, use verifiers, and complete tasks through search and multi-turn interaction. We adopts an analytical viewpoint based on "compression is intelligence": under a fixed task distribution, interface, and compute budget, a stronger agentic system lets a target object be reconstructed with fewer bits. We operationalize the measure with arithmetic coding, seed coding, and a fallback, and evaluate it in five settings: reversed text, chess moves, protein sequences, retrieval-augmented question answering, and semantic story compression; in all of them agentic components reduce codelength. These small, controlled experiments cover component types typical of real agentic systems, show that codelength can analyze how components, observers, and budgets change residual uncertainty, and offer guidance for evaluating real agent systems.