The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence

2026-09-22

AI Summary

研究解决了长上下文模型中近距离背景信息干扰远距离证据的问题，通过引入LYRA机制重新分配注意力，提高对任务相关证据的关注。

Abstract

Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: https://xiaoyuyoung.github.io/LYRA/

Problem

• Proximity Trap
• distant evidence
• proximal background
• context retrieval

Innovation

• LYRA
• t-distributed directional matching
• Proximity Trap
• ProxBench