LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

2026-09-29

AI Summary

This study addresses the limitations of existing long-context evaluations, where accuracy saturation and cost convergence hinder effective differentiation among modern language model frameworks. For the first time, this work establishes efficiency as a critical evaluation dimension, constructing a benchmark platform that jointly assesses effectiveness and efficiency by requiring models to employ diverse retrieval strategies and global-local adaptive reasoning. Methodologically, it integrates techniques such as lexical search and semantic matching to systematically evaluate multiple frontier models and state-of-the-art frameworks. Results demonstrate that even the optimal combination achieves only 68% macro-average accuracy, with substantial efficiency variations observed for identical models across different frameworks. This project introduces a strategic rather than exhaustive evaluation paradigm for long-context processing, revealing the pivotal influence of framework design on practical model performance.

Abstract

Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy--cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model--harness combinations: the best reaches 68\% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.

Problem

• long-context reasoning
• LM harnesses
• benchmark evaluation
• efficiency
• retrieval strategies

Innovation

• Long-Context Reasoning
• LM Harnesses
• Efficiency Evaluation
• Adaptive Retrieval
• Benchmark