🤖 AI Summary
This study addresses the inconsistency in large language models’ behavior during mental health conversations arising from variations in question framing, which undermines their reliability. By constructing multi-frame aligned prompts, the work systematically investigates how contextual framing influences both model outputs and internal representations. It reveals, for the first time, the depth-wise distribution of frame sensitivity within aligned models’ internal representations. Through hierarchical probing, lexical baselines, and targeted activation interventions, the study analyzes the relationship between layer-wise Transformer representations and behavioral outcomes. Findings demonstrate that framing effects are pervasive, that representational information remains decodable throughout the network, and that steering specific representational directions can partially modulate model responses—offering a novel pathway toward enhancing model robustness and controllability.
📝 Abstract
Large language models (LLMs) are increasingly being integrated into mental health support tools and other psychologically sensitive conversational applications. In such settings, behavioral stability and consistency are important for trustworthy human-AI interaction. However, semantically similar concerns can be presented through different contextual framings, potentially eliciting different model responses. Such framing-sensitive variability may challenge user expectations regarding system behavior and complicate the assessment of AI reliability. While prior studies have primarily examined such effects at the behavioral level, less is known about how framing-related variation is reflected in the internal representations of aligned language models. In this work, we investigate these effects using controlled matched prompts spanning multiple contextual framing conditions across several instruction-tuned model families. Across architectures, framing systematically alters interpretive response tendencies. Layer-wise probing analyses show that behavior-associated information remains decodable throughout transformer depth, with architecture-dependent variation in decoding strength. Moreover, held-out framing probes remained consistently above chance across architectures despite strong lexical baselines. Activation steering experiments further suggest that framing-associated representational directions can partially modulate downstream behavioral outcomes. Finally, these findings indicate that robustness to contextual variation may represent an important consideration when evaluating the consistency and trustworthiness of conversational AI systems deployed in mental-health-oriented interactions.