Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcement

2026-09-21

AI Summary

研究解决了视觉-语言模型中因强化学习导致的推理过程模糊问题，通过提出TAME方法抑制模板相关激活，提高过程可监控性。

Abstract

Reinforcement learning (RL) improves reasoning in vision-language models (VLMs) but can induce chain-of-thought (CoT) obfuscation: an operational, non-intentional outcome where task reward or accuracy rises while traces become less grounded and monitorable. Prior work largely documents this decay behaviorally, leaving its representation-level correlates and actionable controls unclear. We find that template- and ground-associated activations become less separable during RL; matched interventions support the contribution of selected features to monitorability degradation. Guided by this evidence, we propose Targeted Anti-obfuscation with Mechanistic Enforcement (TAME), which uses Sparse Autoencoders (SAEs) to combine behavioral feedback with targeted suppression of template-associated activations during RL. Its asymmetric constraint penalizes template activations only above their pre-RL baseline, anchoring the localized features while behavioral feedback promotes grounded refinements. Across VIRL-39k, SPA-VL, and two model families, TAME improves CoT monitorability by up to 30.9 and 16.7 percentage points over Group Relative Policy Optimization (GRPO), respectively. Blinded human evaluation finds higher human monitorability on both datasets, and two held-out monitor families reproduce the monitorability gains. Task accuracy changes are small and mixed, and general-capability benchmarks show task-specific trade-offs. These results provide a path from behavioral monitoring to representation-level oversight for more auditable RL-trained multimodal systems.

Problem

• Reinforcement Learning
• Vision-Language Models
• Chain-of-Thought Obfuscation
• Monitorability

Innovation

• TAME
• Sparse Autoencoders
• Chain-of-thought Obfuscation
• Reinforcement Learning
• Vision-Language Models