Amplifiers or Equalizers? A Longitudinal Study of LLM Evolution in Software Engineering Project-Based Learning

2025-11-28

AI Summary

This study investigates the longitudinal educational impact of large language models (LLMs) in software engineering project-based learning (PBL), focusing on their dual role in shaping learning equity and performance differentiation. Employing a two-year longitudinal cohort design, we integrate quantitative performance analysis with qualitative usage observation to compare the pedagogical efficacy of free (earlier-generation) versus paid (state-of-the-art) LLMs in authentic course projects. Our findings provide the first empirical evidence that LLMs function simultaneously as both “equalizers” and “amplifiers”: they significantly enhance overall practical competency—particularly elevating the average performance of students with weak programming foundations—yet concurrently widen the absolute performance gap between high-achieving and average-performing students. These results offer critical empirical grounding and novel theoretical insights for ethically integrating LLMs into SE education, designing adaptive pedagogical strategies, and informing equitable AI-in-education policy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As LLMs reshape software development, integrating LLM-augmented practices into SE education has become imperative. While existing studies explore LLMs' educational use in introductory programming or isolated SE tasks, their impact in more open-ended Project-Based Learning (PBL) remains unexplored. This paper introduces a two-year longitudinal study comparing a 2024 (using early free LLMs, $n$=48) and 2025 (using the latest paid LLMs, $n$=46) cohort. Our findings suggest the latest powerful LLMs' dual role: they act as "equalizers," boosting average performance even for programming-weak students, providing opportunities for more authentic SE practices; yet also as "amplifiers," dramatically widening absolute performance gaps, creating new pedagogical challenges for addressing educational inequities.

Problem

• Investigating LLM impact on software engineering project-based learning outcomes
• Comparing early free versus latest paid LLMs across two student cohorts
• Analyzing how LLMs act as equalizers and amplifiers in education

Innovation

• Two-year longitudinal study comparing LLM cohorts
• Analyzing LLMs as equalizers and amplifiers in education
• Investigating LLM impact on project-based learning outcomes