🤖 AI Summary
Current evaluations of large language models predominantly rely on knowledge recall or abstract reasoning, failing to adequately assess their capacity to integrate multiple cognitive abilities in real-world tasks. To address this gap, this work proposes the Grounded Integration Measure (GIM) benchmark, comprising 820 original items whose core difficulty stems from the integration of multiple cognitive domains—such as constraint satisfaction, state tracking, cognitive vigilance, and audience adaptation—for the first time. GIM incorporates expert-authored questions, fine-grained scoring, and a public-private item design, and leverages two-parameter logistic item response theory (2PL IRT) to calibrate over 200,000 model responses across 28 models, effectively mitigating accuracy distortion. Based on this framework, we release a comprehensive leaderboard covering 22 models and 47 configurations, and large-scale experiments reveal that test-time compute allocation influences performance as significantly as model selection itself.
📝 Abstract
As LLM benchmarks saturate, the evaluation community has pursued two strategies to increase difficulty: escalating knowledge demands (GPQA, HLE) or removing knowledge entirely in favor of abstract reasoning (ARC-AGI). The first conflates memorization with capability; the second divorces reasoning from the practical contexts in which it matters. We take a different approach. The Grounded Integration Measure (GIM) is a benchmark of 820 original problems (615 public, 205 private) where difficulty comes from integration; individual problems require coordinating multiple cognitive operations (constraint satisfaction, state tracking, epistemic vigilance, audience calibration) over broadly accessible knowledge, so that reasoning stays grounded in realistic tasks without being gated on specialized expertise. Each problem is an original expert-authored composition, majority with rubric-decomposed scoring (median 6 independently judged criteria). A balanced public--private split provides built-in contamination diagnostic. We calibrate a continuous response 2-parameter logistic (2PL) IRT model over >200k prompt-response pairs across 28 models, producing robust ability estimates that correctly order test-configurations even when raw accuracy is distorted by errors or missing data, addressing a common challenge in benchmark reporting. Using this framework, we present a comprehensive leaderboard spanning 22 models and 47 test-configurations (unique model, thinking-level pairs), and conduct what is to our knowledge the most extensive published study of how test-time compute trades off against model capability on a fixed benchmark: 11 models swept across 35 test-configurations. We observe that within-family configuration choices, such as thinking budget and quantization, matter as much as model selection. We release the evaluation framework, calibrated IRT parameters, and all public problems.