The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys

2026-07-26

AI Summary

This study presents the first systematic evaluation of the impact of AI-generated fraudulent responses on both quantitative and qualitative conclusions in software engineering surveys. Through secondary analysis of four existing survey datasets, the authors integrate manual identification of suspicious responses, AI-generated text detection tools, descriptive statistics, and thematic analysis to compare findings before and after data cleaning. Results indicate that quantitative outcomes remain largely robust, though moderate shifts emerge in certain demographic and analytical variables. In contrast, qualitative conclusions exhibit notable interpretive deviations, influenced by contextual framing, coding salience, and the nature of evidentiary support. These findings underscore the critical need for multimethod triangulation in studies incorporating open-ended responses to ensure validity and reliability.

Abstract

Background: Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys. Aims: This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings. Method: We conducted a secondary analysis of four software engineering survey datasets using manual identification of suspicious responses, automated AI generated text detection, descriptive statistical analysis, and thematic analysis. We compared findings obtained from the original and manually cleaned datasets. Results: Quantitative findings generally remained stable after filtering suspicious responses, although some demographic and analytical variables showed moderate variation, affecting the interpretation of specific participant groups and contextual characteristics. In contrast, qualitative findings were more strongly influenced by changes in contextual framing, code prominence, and the nature of the evidence supporting interpretation, shaping how participants' experiences and study contexts were interpreted and characterized. Conclusions: AI assisted participation may influence software engineering survey findings differently depending on the type of analysis being conducted. The findings reinforce the importance of combining multiple validation procedures, particularly in studies relying on open ended responses.

Problem

• AI-generated responses
• software engineering surveys
• survey validity
• fraudulent participation
• response authenticity

Innovation

• AI-generated responses
• survey validity
• software engineering
• qualitative analysis
• fraud detection