Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression

2026-08-02

AI Summary

This work addresses a critical limitation in existing KV cache compression methods, which are typically evaluated solely on final answer accuracy while neglecting the fidelity of the reasoning process. This oversight can lead to an “answer-evidence gap,” wherein correct answers lack coherent reasoning support. To tackle this issue, the authors propose a fixed reasoning trajectory replay protocol, formally defining and quantifying the phenomenon for the first time. They introduce two novel metrics—answer chain consistency and perturbation faithfulness—and systematically evaluate ten token pruning methods alongside one quantization approach across diverse tasks, including mathematical reasoning, scientific question answering, clinical calculation, and long-context retrieval. Results reveal that while most token pruning techniques preserve high answer accuracy, they significantly degrade reasoning consistency; in contrast, coverage-preserving quantization minimally disrupts the reasoning process, demonstrating superior fidelity under compression.

Abstract

KV cache compression is commonly evaluated by final-answer accuracy, implicitly assuming that preserving the answer also preserves the reasoning that supports it. We test this assumption for large reasoning models and show that it can fail: under compression, correct answers and the validity of their visible supporting rationales can be preserved at different rates. We study this failure with a controlled fixed-trace replay protocol, which holds reasoning content fixed and isolates whether compression preserves usable information from an already available trace. We evaluate ten token-eviction KV compression methods and one quantization method on three models across mathematical reasoning, scientific QA, clinical calculation, and long-context retrieval. We measure final accuracy, answer-chain consistency, and perturbation faithfulness. Across tasks, token-eviction methods can preserve competitive final-answer accuracy while substantially degrading chain support or perturbation faithfulness. We call this the answer-evidence gap. A coverage-preserving quantization control is substantially less affected, suggesting that the failure is tied less to KV memory reduction itself than to losing access to parts of the reasoning trace. Code is available at https://github.com/famous-blue-raincoat/Safe_KV_Compress.

Problem

• KV cache compression
• reasoning faithfulness
• answer-evidence gap
• large reasoning models
• model interpretability

Innovation

• KV cache compression
• reasoning faithfulness
• answer-evidence gap
• fixed-trace replay
• perturbation faithfulness