Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

2026-09-25

AI Summary

This study addresses the inefficiency of large language models (LLMs) in reasoning over long contexts, where critical evidence is often sparse and obscured by redundancy. To this end, it proposes H2S, a compress-then-reason paradigm built upon a three-stage "Identify-Integrate-Generate" framework. The method leverages process-level reward reinforcement learning to train models to explicitly select and integrate essential evidence, complemented by a purpose-built dataset and evaluation benchmark. Experimental results demonstrate that a 14B-parameter model operating under 128K-token inputs retains 97.1% of its performance while outperforming larger-scale counterparts. These findings indicate that H2S substantially enhances the long-context comprehension and reasoning capabilities of LLMs without requiring proportional increases in model scale.

Abstract

Long-context understanding requires large language models (LLMs) to reason over lengthy documents, conversations, and code, yet task-relevant evidence is often sparse and scattered amid substantial irrelevant and redundant content. We propose Highlight-Then-Summarize (H2S), a compress-then-reason paradigm that first identifies source-grounded, question-relevant evidence and then integrates it into a compact, question-conditioned summary before producing the final answer. To train this behavior, we construct H2S-Dataset, comprising 6,647 examples from 11 benchmark families with an average context length of 43.9K tokens, and introduce H2S-RL, which provides process-level rewards for evidence selection and summary construction in addition to final-answer correctness. We evaluate on H2S-Bench, a seven-task long-context suite. Under a shared 128K input and 4K output budget, H2S-14B achieves an average score of 32.60, outperforming Qwen3.8-27B by 10.17 points and obtaining the strongest overall result among the evaluated open-source models. H2S-14B also achieves the highest Evidence-Summary Quality score and retains 97.1% of its 16K-budget performance with only a 4K output budget. These results show that explicitly selecting and integrating evidence improves long-context reasoning while enabling more compact generation.

Problem

• Long-context understanding
• Evidence compression
• Large language models
• Sparse evidence

Innovation

• Highlight-Then-Summarize
• Long-context understanding
• Process-level reinforcement learning
• Evidence compression
• Compress-then-reason