How Software Engineering Students Use LLMs to Write Research Papers: An Experience Report

2026-06-03

AI Summary

This study investigates how software engineering students employ large language models (LLMs) during the research paper writing process, with a focus on usage patterns, encountered challenges, and educational implications. In a third-year software architecture course, students completed short papers using rapid or gray literature review methodologies and submitted LLM usage disclosure statements. Through a systematic analysis of 146 such statements—combining LLM-assisted categorization with manual cross-validation—the study reveals, for the first time, that students extensively utilize LLMs across multiple stages, including brainstorming, clarifying methodology, organizing results, and polishing writing. Notably, students consistently expressed concern about the accuracy and verifiability of LLM-generated content. These findings offer empirical evidence and pedagogical insights for the effective integration of LLMs into software engineering education.

Abstract

Large language models are increasingly becoming part of software engineering education, including activities involving empirical software engineering and evidence synthesis. This paper reports an educational experience involving the integration of reflective LLM use into an empirical methods assignment in a third-year software architecture course. Students were asked to develop a short research paper using either a rapid review or a gray literature review methodology and to disclose how LLMs were used throughout the assignment. We analyzed 146 student disclosure statements using a cross-analysis process combining LLM-assisted categorization with manual verification and refinement by the researchers. The reflections describe how students incorporated LLMs during activities such as brainstorming, methodological clarification, organization of findings, and writing refinement, while also reporting concerns regarding inaccuracies and verification of generated content. This experience report discusses lessons learned and educational implications for integrating AI-assisted technologies into empirical software engineering education.

Problem

• large language models
• software engineering education
• empirical software engineering
• student writing
• AI-assisted learning

Innovation

• reflective LLM use
• empirical software engineering education
• gray literature review
• LLM-assisted categorization
• AI-integrated pedagogy