Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

2026-07-22

AI Summary

Current natural language autoencoders rely on reconstruction scores to evaluate explanation faithfulness, which often fails to detect individual false claims, allowing high-scoring explanations to still contain hallucinated content. This work proposes RECAP, a method that jointly trains a linear auxiliary prediction head alongside the target model, introducing a decodability-based supervision mechanism that enables external probes to independently verify internal representations. We design two auditing protocols—grounded-vs-true crossover and evaluator swapping—to detect spurious encodings. Experiments on Qwen and Pythia models demonstrate RECAP’s effectiveness: probe-based decoding of specified content achieves an AUC of 0.96, substantially outperforming the baseline (0.82). Moreover, even when faced with adversaries deliberately manipulating explanations, RECAP reliably identifies falsehoods (AUC 0.95 versus 0.51 in the control condition).

Abstract

Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims: if flipping a claim does not change the reconstruction, the claim is never penalized. We show the test is passed in two ways, neither faithful. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent, so the score tracks gist, not specific facts. Under exact synthetic ground truth, the standard recipe develops co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, and fixes that leave the target model unchanged do not help. We contribute two audit protocols, the grounded-vs-true cross and the evaluator swap, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors): linear heads trained alongside the target model to keep designated content decodable. On RECAP-trained sandbox models, fresh verbalizers state the designated content truly and the codes vanish, at a +0.001-nat cost. This replicates on a pretrained Pythia-160M: the content becomes reliably probe-decodable, though a fresh verbalizer conveys it only in part (truth 0.44-0.46 vs a near-zero control). For interpretability, high reconstruction does not certify individual claims. For AI safety, RECAP makes designated internal content independently checkable against probes rather than asserted by prose a model can game: an independent probe scores the verbalizer's true claims above its false ones (AUC 0.96, vs 0.82 without RECAP). Against an adversary that edits an explanation to maximize the reconstruction score while lying (suppressing ~87% of its lie penalty), the RECAP probe still flags the lies (AUC 0.95) while the control probe collapses to chance (0.51).

Problem

• activation explanations
• faithfulness
• reconstruction
• decodability
• AI safety

Innovation

• RECAP
• decodability supervision
• activation explanations
• probing
• verifiable interpretability