Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research

2025-06-15

AI Summary

Large language models (LLMs) exert dual influences on software engineering (SE) research—accelerating ideation and automation while simultaneously undermining interpretability and distorting methodological rigor. Method: Drawing on McLuhan’s Four Laws of Media, this study systematically analyzes LLMs’ fourfold effects in SE research: enhancement, obsolescence, retrieval, and reversal. Integrating human-centered AI design principles with consensus-building from interdisciplinary workshops, it proposes a human-driven, AI-augmented research governance framework centered on the synergistic interplay of “instrumentalization” (LLMs as tools) and “objectification” (LLMs as research subjects). Contribution/Results: The work yields an actionable community initiative and practical guidelines for SE researchers, offering both theoretical grounding and operational pathways to uphold scientific rigor, ethical accountability, and model interpretability in LLM-augmented research practice.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The adoption of Large Language Models (LLMs) is not only transforming software engineering (SE) practice but is also poised to fundamentally disrupt how research is conducted in the field. While perspectives on this transformation range from viewing LLMs as mere productivity tools to considering them revolutionary forces, we argue that the SE research community must proactively engage with and shape the integration of LLMs into research practices, emphasizing human agency in this transformation. As LLMs rapidly become integral to SE research - both as tools that support investigations and as subjects of study - a human-centric perspective is essential. Ensuring human oversight and interpretability is necessary for upholding scientific rigor, fostering ethical responsibility, and driving advancements in the field. Drawing from discussions at the 2nd Copenhagen Symposium on Human-Centered AI in SE, this position paper employs McLuhan's Tetrad of Media Laws to analyze the impact of LLMs on SE research. Through this theoretical lens, we examine how LLMs enhance research capabilities through accelerated ideation and automated processes, make some traditional research practices obsolete, retrieve valuable aspects of historical research approaches, and risk reversal effects when taken to extremes. Our analysis reveals opportunities for innovation and potential pitfalls that require careful consideration. We conclude with a call to action for the SE research community to proactively harness the benefits of LLMs while developing frameworks and guidelines to mitigate their risks, to ensure continued rigor and impact of research in an AI-augmented future.

Problem

• Analyzing LLM impact on software engineering research methods
• Ensuring human oversight in AI-augmented SE research
• Balancing LLM benefits and risks for scientific rigor

Innovation

• Using LLMs for accelerated ideation and automation
• Applying human-centric oversight for ethical research
• Analyzing LLM impact via McLuhan's Tetrad framework