Probing the Stochastic Machine: Engaging with LLMs in Statistics Curricula Through Veridical Data Science

2026-06-28

AI Summary

This work addresses the stochasticity, biases, and prompt sensitivity inherent in large language models (LLMs) by treating them not merely as tools but as objects of systematic statistical inquiry. It proposes a novel pedagogical framework that integrates LLMs’ random behavior into statistics curricula across four levels—from introductory to graduate—grounded in the Veridical Data Science paradigm and the Predictability–Computability–Stability (PCS) principles. Through structured activities such as experimental design, output distribution analysis, and stability audits, students engage in empirical investigations to critically assess LLM behavior and limitations in data analysis contexts. This approach fosters a deeper, evidence-based understanding of the reliability and constraints of LLMs, equipping learners to evaluate their use in statistical practice with rigor and skepticism.

Abstract

Large language models (LLMs) are interactive stochastic systems whose most consequential behaviors are still only partially understood. This discussion argues that statistics curricula should treat LLMs not only as tools, but as objects of inquiry: students can probe variability, bias, and prompt sensitivity by designing small experiments and analyzing distributions of outputs. Building on the Veridical Data Science framework and Predictability-Computability-Stability (PCS) principles, this discussion outlines how to organize critical LLM engagement across educational levels and propose four curricular examples, from introductory ``ask it twice'' activities to graduate PCS stability audits of LLM-based analysis workflows.

Problem

• large language models
• statistics education
• stochastic systems
• prompt sensitivity
• veridical data science

Innovation

• large language models
• Veridical Data Science
• Predictability-Computability-Stability
• stochastic systems
• statistics education