Learning to Code with Context: A Study-Based Approach

2025-12-04

AI Summary

Generative AI poses significant challenges to software engineering education, necessitating strategies that preserve foundational development competencies while fostering responsible AI tool usage. Method: We conducted an empirical study in which students collaboratively developed games, systematically analyzing AI’s efficacy and practical challenges across requirements analysis, coding, debugging, and documentation. To address limitations of generic AI assistants, we designed a project-context-aware, localized LLM assistant integrating retrieval-augmented generation (RAG) for fine-grained, explainable codebase and documentation understanding. Contribution/Results: Our context-aware assistant significantly improved task completion efficiency and code quality, mitigated hallucination and overreliance, and enabled identification of recurrent AI usage patterns and pedagogically grounded risk-mitigation strategies in software engineering education.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

The rapid emergence of generative AI tools is transforming the way software is developed. Consequently, software engineering education must adapt to ensure that students not only learn traditional development methods but also understand how to meaningfully and responsibly use these new technologies. In particular, project-based courses offer an effective environment to explore and evaluate the integration of AI assistance into real-world development practices. This paper presents our approach and a user study conducted within a university programming project in which students collaboratively developed computer games. The study investigates how participants used generative AI tools throughout different phases of the software development process, identifies the types of tasks where such tools were most effective, and analyzes the challenges students encountered. Building on these insights, we further examine a repository-aware, locally deployed large language model (LLM) assistant designed to provide project-contextualized support. The system employs Retrieval-Augmented Generation (RAG) to ground responses in relevant documentation and source code, enabling qualitative analysis of model behavior, parameter sensitivity, and common failure modes. The findings deepen our understanding of context-aware AI support in educational software projects and inform future integration of AI-based assistance into software engineering curricula.

Problem

• Investigates generative AI use in software development education
• Examines AI tool effectiveness across development phases and tasks
• Analyzes context-aware AI support for project-based learning

Innovation

• Used Retrieval-Augmented Generation for contextual support
• Deployed a local LLM assistant with repository awareness
• Analyzed model behavior and failure modes in education