Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safety

2025-10-20

AI Summary

Existing text-based safety analysis methods for reasoning often overlook subtle harmful patterns and fail to capture latent unsafe reasoning paths within large language models (LLMs). To address this, we introduce the first sentence-level fine-grained annotated dataset of reasoning-chain safety behaviors, enabling precise temporal localization of safe or harmful behavior during LLM generation. Methodologically, we integrate sentence-level behavioral annotation, layer-wise activation vector extraction, and steering vectors to achieve the first detectable and intervenable representational modeling of safety behavior within reasoning chains. Experiments demonstrate that the learned representations effectively identify and modulate internal safety-related activations, significantly improving the accuracy and controllability of safety monitoring during inference. Our results validate the effectiveness and feasibility of activation-state-based, fine-grained safety supervision as a novel paradigm for reasoning safety.

Technology Category

• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

Recent work has highlighted the importance of monitoring chain-of-thought reasoning for AI safety; however, current approaches that analyze textual reasoning steps can miss subtle harmful patterns and may be circumvented by models that hide unsafe reasoning. We present a sentence-level labeled dataset that enables activation-based monitoring of safety behaviors during LLM reasoning. Our dataset contains reasoning sequences with sentence-level annotations of safety behaviors such as expression of safety concerns or speculation on user intent, which we use to extract steering vectors for detecting and influencing these behaviors within model activations. The dataset fills a key gap in safety research: while existing datasets label reasoning holistically, effective application of steering vectors for safety monitoring could be improved by identifying precisely when specific behaviors occur within reasoning chains. We demonstrate the dataset's utility by extracting representations that both detect and steer safety behaviors in model activations, showcasing the potential of activation-level techniques for improving safety oversight on reasoning. Content Warning: This paper discusses AI safety in the context of harmful prompts and may contain references to potentially harmful content.

Problem

• Detecting subtle harmful patterns in chain-of-thought reasoning
• Monitoring safety behaviors through model activation analysis
• Identifying specific unsafe reasoning steps within AI processes

Innovation

• Dataset enables activation-based safety behavior monitoring
• Sentence-level annotations extract steering vectors for detection
• Activation-level techniques detect and steer safety behaviors