Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study

2026-05-08

AI Summary

This study investigates the reliability of large language models (LLMs) in replicating human reasoning for qualitative coding of psychological safety in software engineering communities, with a focus on performance disparities and systematic biases across different prompting strategies. Through controlled experiments, the authors evaluate Cohen’s κ agreement and stability of Claude Haiku, DeepSeek-Chat, and Gemini 2.5 Flash under zero-shot and few-shot closed-ended prompting. The work presents the first systematic quantification of few-shot prompting effects on LLM-based qualitative coding, revealing that this strategy significantly improves Claude Haiku’s intercoder agreement (Δκ = +0.034). Claude Haiku and DeepSeek-Chat demonstrate the highest stability (SD ≈ 0.017). All models consistently over-predict “sharing negative feedback” and under-predict “expressing concerns,” offering empirical insights and methodological guidance for LLM-assisted qualitative research.

Abstract

Qualitative analysis plays a pivotal role in understanding the human and social aspects of software engineering. However, it remains a demanding process shaped by the subjective interpretation of individual researchers and sensitive to methodological choices such as prompt design. Recent advancements in Large Language Models (LLMs) offer promising opportunities to support this type of analysis, although their reliability in reproducing human qualitative reasoning under varying prompting conditions remains largely untested. This study presents a controlled empirical evaluation of three LLMs -- Claude Haiku, DeepSeek-Chat, and Gemini 2.5 Flash -- across two prompt engineering strategies (zero-shot and multi-shot closed coding), using Cohen's kappa as the primary agreement metric over ten independent runs per configuration. Results suggest that multi-shot prompting significantly improves agreement for Claude Haiku (Delta kappa = +0.034, Wilcoxon p = 0.004) but not for DeepSeek-Chat or Gemini 2.5 Flash. Intra-model stability varies substantially -- DeepSeek-Chat and Claude Haiku exhibit the lowest variance (SD approx. 0.017), while Gemini 2.5 Flash is the least stable (SD = 0.038). A systematic over-prediction of "Sharing Negative Feedback" is identified across all models (bias ratios up to 5.25x), alongside consistent under-prediction of "Expressing Concerns." Collectively, these findings provide empirical evidence for prompt engineering guidelines in LLM-assisted qualitative coding for software engineering research.

Problem

• qualitative coding
• psychological safety
• prompt engineering
• Large Language Models
• software engineering

Innovation

• prompt engineering
• large language models
• qualitative coding
• psychological safety
• empirical evaluation