Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

2026-08-05

AI Summary

This work addresses the inefficiency of existing KV cache compression methods, which apply uniform strategies that ignore the varying tolerance for context loss across inference steps and overlook the feedback effect of compression on generation length—leading to high computational costs in large reasoning models due to excessively long chains of thought. To mitigate this, we propose ReCo, a novel framework that introduces, for the first time, a lightweight process reward estimator to dynamically assess the importance of each reasoning step. ReCo jointly optimizes three mechanisms: reward-guided adaptive KV cache compression, reward-segmented reflection token penalization, and confidence-driven early stopping. Experiments across three models and six benchmarks show that ReCo reduces generated tokens by 37%–65% and achieves 2.08–2.35× lower end-to-end latency compared to full chain-of-thought inference, while largely preserving accuracy.

Abstract

Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache. Two observations point the other way. First, a reasoning state's tolerance to context loss varies along the trajectory, and process reward tracks it: deleting tokens at high-reward steps preserves accuracy far better than deleting the same budget at random. Second, compression is not free on the generation side, since a smaller cache leads the model to generate more tokens, partly canceling the saving. Together these motivate coordinating both sides under a single process reward. We propose ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: (1) reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, (2) a reward-banded penalty on reflection tokens that curbs redundant generation, and (3) confidence-based early stopping that triggers when the reasoning is reliable. Across three reasoning models and six benchmarks, ReCo reduces generated tokens by 37%-65% and end-to-end latency by 2.08x-2.35x over Full CoT, all while largely preserving accuracy.

Problem

• efficient reasoning
• KV-cache compression
• chain-of-thought
• inference cost
• overthinking

Innovation

• reward-coordinated compression
• KV-cache compression
• adaptive reasoning
• process reward
• early stopping