Structured AI Demonstrations and Student LLM Use in Engineering Mechanics: Study Design and Preliminary Results

2026-07-30

AI Summary

This study addresses the lack of domain-specific empirical research on students’ use of large language models (LLMs) in engineering mechanics education, which hinders the development of effective pedagogical policies. Conducted during the Spring 2026 course offering, the project systematically investigates students’ LLM usage patterns, validation behaviors, and their relationship to academic performance through surveys, learning analytics, and nine structured, instructor-led AI demonstrations. The work proposes and openly releases a comprehensive empirical framework comprising reproducible survey instruments, instructional interventions, and AI demonstration materials. Preliminary findings indicate evolving student engagement with AI and a nuanced relationship between AI reliance and academic outcomes. The primary contribution lies in providing the engineering education community with a reusable methodological toolkit to foster collaborative, evidence-based inquiry into the impact of LLMs.

Abstract

The rapid integration of large language models (LLMs) into undergraduate education presents an urgent challenge for engineering instructors. Despite widespread student adoption, there remains a critical lack of domain-specific empirical evidence to guide pedagogical policies and classroom interventions. This manuscript presents a descriptive study design and preliminary findings from an undergraduate engineering mechanics course conducted in Spring 2026. We detail a reproducible survey instrument used to capture student AI usage patterns, attitudes, and verification practices, which are subsequently linked to academic performance metrics. Additionally, we document a deployable sequence of nine structured, instructor-led AI demonstrations designed to model strategic LLM delegation and evaluation. While our preliminary data highlight shifting student behaviors and complex relationships between AI reliance and course outcomes, the primary contribution of this work is the provision of an open-access methodological framework. By making our complete study design, survey tools, and demonstration materials publicly available, we urge other engineering educators to collect and share similar empirical data. Navigating this unprecedented technological shift will require a collaborative, evidence-based approach to fully understand its long-term impacts on student learning.

Problem

• large language models
• engineering education
• student AI use
• pedagogical policy
• empirical evidence

Innovation

• structured AI demonstrations
• LLM integration in engineering education
• empirical study design
• student AI usage patterns
• open-access pedagogical framework