No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow

2026-09-24

AI Summary

This study addresses the limitation that existing evaluations of long-context models predominantly focus on simple tasks whose difficulty scales linearly with corpus length, thereby neglecting high-complexity reasoning challenges. To bridge this gap, this work introduces the concept of Corpus Task Complexity (CTC) and constructs CTC-Bench, a benchmark comprising ten tasks exhibiting quadratic or higher-order complexity growth. Through this benchmark, we systematically evaluate the long-range reasoning performance of full-attention, block-sparse, and hybrid attention mechanisms. Our findings reveal that efficient attention mechanisms suffer significant degradation on high-complexity tasks, correcting overly optimistic conclusions derived from low-complexity evaluations. Furthermore, this work demonstrates that full attention remains indispensable for large-scale, high-complexity reasoning, clearly identifying it as the core bottleneck in current long-context scaling efforts.

Abstract

Given a large corpus, the questions one might ask can vary -- from "When was the first human heart transplant?" to "What are all the contradictory claims in this literature?" -- but what makes some questions more challenging than others? In this work, we define a notion of Corpus Task Complexity (CTC) that characterizes tasks by how their difficulty grows with corpus size; for instance, a retrieval query only requires a single linear pass over a corpus, while finding contradictions requires checking a quadratically growing set of claim pairs. Observing that prior work has largely only studied tasks whose difficulty grows linearly with corpus size, which we call low CTC tasks, we introduce 10 new tasks belonging to a class of high CTC whose difficulty grows quadratically or more in corpus size. We find that high-CTC tasks not only grow much more challenging on average at longer contexts for LCLMs, they reverse many modeling conclusions drawn solely from low-CTC evaluations. For instance, efficient block-sparse and hybrid attention approaches consistently match full attention performance on low-CTC tasks, but degrade much more on high-CTC tasks. Large-corpus high-CTC reasoning thus remains an open challenge as full attention is too costly to scale, motivating future research on these tasks. We release our code, data, and 22-task suite (CTC-Bench), to facilitate future research in this area.

Problem

• Corpus Task Complexity
• Long-context Language Models
• High CTC Tasks
• Attention Mechanisms
• Reasoning

Innovation

• Corpus Task Complexity
• Long-context Language Models
• Attention Mechanisms
• Benchmark Suite
• Task Difficulty Scaling