Mergeable Model-Side Aggregation States for Long-Context Language Models

2026-07-28

AI Summary

This work addresses the significant performance degradation of long-context language models on non-additive set aggregation tasks—such as cardinality estimation and set relations—as context length increases. The authors propose the first integration of mergeable, compact HyperLogLog (HLL) sketch states into a frozen language model, enabling real-time HLL state updates via identity extraction and hashing. This approach supports efficient cross-segment merging and direct state reading, eliminating the need for generate-execute-return loops. The method achieves constant memory overhead independent of both context length and set cardinality, yielding an average relative error of only 1.6% in million-scale cardinality estimation. Merged results across 256 segments match those of a single full pass, and the system attains 99.2% accuracy across 3,969 aggregation tasks—surpassing direct reasoning and chain-of-thought baselines by over 56 and 60 percentage points, respectively.

Abstract

A known limitation of long-context language models is their increasingly unreliable performance in non-additive, set-based aggregation as context length grows. Examples include cardinality estimation, set relationships, and grouped statistics, which widely exist in logs, program outputs, tables, and multi-turn conversations. To provide the aggregation state required by these tasks, we introduce a model-side aggregation interface that maintains compact Hash-based HyperLogLog (HLL) sketch states alongside a frozen language model. While the model processes the context, an extractor maps each relevant record to a canonical identity. The identity is then hashed and updates the HLL state. These states can be merged across context segments and/or read out directly for downstream reasoning, avoiding an additional generate-execute-return cycle. We validate the proposed approach by setting the HLL state size as 2 KiB (2,048 registers), which does not increase with context length or set cardinality. In a distinct-count experiment involving one million records, the mean relative error was 1.6%. In a separate merge test, states built from as many as 256 segments produced exactly the same readout as a single pass over the same stream. On 3,969 aggregate-then-reason tasks from 174 source windows, the fixed-budget interface reached 99.2% accuracy on Gemma 4 (31B, BF16), compared with 100.0% under exact aggregation; the paired gap was 0.8 percentage points (95% window-cluster CI: 0.5-1.3 points). On a matched set of 174 items, our method improved over direct full-context reasoning by 63.2 points on Qwen and 56.3 points on Gemma. The corresponding gains over chain-of-thought (CoT) reasoning were 60.9 and 63.2 points, respectively. On a fixed 1,200-task Oolong-Synth subset, our method reached 91.1% on Qwen and 99.3% on Gemma. Code is available at https://github.com/songdc98/sketchops.

Problem

• long-context language models
• set-based aggregation
• cardinality estimation
• aggregation reliability
• non-additive aggregation

Innovation

• model-side aggregation
• HyperLogLog sketch
• long-context language models
• mergeable states
• set-based reasoning