KV-streams for Efficient Compaction in Agentic Reinforcement Learning

2026-09-28

AI Summary

This study addresses the GPU memory bottleneck caused by long contexts in agent reinforcement learning and the throughput limitations of conventional compression strategies. To this end, it proposes KV-streams, a plug-and-play method that achieves efficient context compression by streaming rather than resetting the KV cache. Furthermore, this work demonstrates for the first time that reinforcement learning alone can elicit the emergent ability to leverage streamed KV caches as recurrent states, without requiring additional supervision. Experimental results indicate that this strategy yields a 2.6× to 5× training speedup across three compression settings while preserving model performance.

Abstract

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

Problem

• Agentic Reinforcement Learning
• Context Compaction
• KV Cache
• Training Throughput
• Large Language Models

Innovation

• KV-streams
• Context Compaction
• Agentic Reinforcement Learning
• KV Cache Streaming
• Recurrent State