Students' Practices and Skills in the LLM-Era: "You Can't Outsource the Struggle and Still Get the Skill"

2026-07-31

AI Summary

This study addresses the tendency of software engineering graduate students to excessively outsource cognitive tasks to large language models (LLMs) during research, which often results in underdeveloped research competencies and suboptimal outcomes. Drawing on a qualitative content analysis of 1,383 posts from five research-oriented Reddit communities, the work systematically identifies and articulates critical research skills that students frequently neglect when collaborating with LLMs. The findings reveal prevailing patterns of overreliance on LLMs and their detrimental impact on scholarly development. Notably, this research is the first to explicitly define the collaborative intelligence literacy essential for graduate students in the AI era, offering both empirical evidence and theoretical grounding for redesigning graduate curricula to better support human–AI collaborative research practices.

Abstract

Generative AI tools have been rapidly learned in the daily workflow of graduate students in Software Engineering, but little is known about what AI-related skills they actually need for effective use in empirical research. Without this understanding, graduate programs cannot prepare students to conduct rig-orous research in the LLM era, risking creating a generation of researchers who delegate tasks without the necessary expertise. By analyzing 1,383 posts from five research-focused subreddits, we found that students systematically outsource the cognitive effort required to develop research skills and end up with neither the expected results nor the necessary competence. Naming these missing skills is the first step toward curricula that teach graduate students to work \emph{with} LLMs without being replaced by them.

Problem

• Generative AI
• LLM
• research skills
• cognitive outsourcing
• graduate education

Innovation

• generative AI
• large language models
• research skills
• cognitive outsourcing
• graduate education