Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses

2026-09-22

AI Summary

研究调查了学生在数据科学课程中使用大型语言模型的行为，并评估了由AI生成的多项选择题难度的有效性，发现AI预测的难度与实际不符。

Abstract

This paper presents a multi-source classroom study conducted during a 10-week quarter in data science courses at Drexel University. We first investigate the behaviors of student engagement with large language models (LLMs) using four surveys across three data science courses. Second, we evaluate the construct validity of multiple-choice questions (MCQs) generated by an LLM for in-lecture retrieval practice. Based on 378 authored questions (311 deployed, producing 7{,}888 student responses), we analyze whether the difficulty ratings assigned by an LLM match empirical item difficulty. Our study shows that student engagement with LLMs varied across courses and increased over the term. Although students expressed high satisfaction and reported saving considerable time, their perception of deep learning benefits declined, and many noted a tendency toward over-reliance. Regarding the difficulty ratings of LLM-generated MCQs, the Easy, Medium, and Hard labels correlated closely with its assigned Bloom's Taxonomy levels (Spearman $ρ=0.90$), reflecting an artifact of co-generation. However, neither metric predicted empirical item difficulty (difficulty label $ρ=0.06$; Bloom level $ρ=0.02$). The ratings reflect the structural formatting of a question rather than its underlying difficulty.

Problem

• Large Language Models
• Student Engagement
• Multiple-Choice Questions
• Question Difficulty

Innovation

• large language models
• student engagement
• multiple-choice questions
• Bloom's Taxonomy
• empirical item difficulty