Technology Category
Machine Learning: Large Multimodal Models (LMMs)Humans and AI: AI for AccessibilityNatural Language Processing: Safety and Robustness
Automated scoring of the Ambiguous Intentions Hostility Questionnaire using fine-tuned large language models
🤖 AI Summary
Manual scoring of open-ended responses in the Ambiguous Intentions Hostility Questionnaire (AIHQ) is time-intensive and laborious. Method: We propose the first large language model (LLM)-based automated scoring framework for the AIHQ, employing supervised fine-tuning to identify two core psychological constructs—hostile attribution bias and aggressive response bias—integrated with fine-grained semantic analysis and high-quality human-annotated data. The resulting dual-mode system supports both local and cloud deployment. Contribution/Results: Our method achieves high inter-rater agreement with human scorers across diverse contexts and populations—including individuals with traumatic brain injury and healthy controls (r > 0.90)—demonstrating robust generalizability. It successfully replicates clinically meaningful group differences and validates efficacy on an independent nonclinical dataset. This work represents the first application of fine-tuned LLMs to AIHQ automated assessment, substantially enhancing the efficiency, scalability, and accessibility of psychological evaluation.
Technology Category
Application Category
📝 Abstract
Hostile attribution bias is the tendency to interpret social interactions as intentionally hostile. The Ambiguous Intentions Hostility Questionnaire (AIHQ) is commonly used to measure hostile attribution bias, and includes open-ended questions where participants describe the perceived intentions behind a negative social situation and how they would respond. While these questions provide insights into the contents of hostile attributions, they require time-intensive scoring by human raters. In this study, we assessed whether large language models can automate the scoring of AIHQ open-ended responses. We used a previously collected dataset in which individuals with traumatic brain injury (TBI) and healthy controls (HC) completed the AIHQ and had their open-ended responses rated by trained human raters. We used half of these responses to fine-tune the two models on human-generated ratings, and tested the fine-tuned models on the remaining half of AIHQ responses. Results showed that model-generated ratings aligned with human ratings for both attributions of hostility and aggression responses, with fine-tuned models showing higher alignment. This alignment was consistent across ambiguous, intentional, and accidental scenario types, and replicated previous findings on group differences in attributions of hostility and aggression responses between TBI and HC groups. The fine-tuned models also generalized well to an independent nonclinical dataset. To support broader adoption, we provide an accessible scoring interface that includes both local and cloud-based options. Together, our findings suggest that large language models can streamline AIHQ scoring in both research and clinical contexts, revealing their potential to facilitate psychological assessments across different populations.
Problem
Research questions and friction points this paper is trying to address.
Automate scoring of hostile attribution bias in AIHQ responses
Reduce time-intensive human rating for open-ended social bias questions
Validate large language models for clinical and research assessments
Innovation
Methods, ideas, or system contributions that make the work stand out.
Fine-tuned large language models for AIHQ scoring
Automated hostility and aggression response ratings
Accessible local and cloud-based scoring interface
🔎 Similar Papers
No similar papers found.
