AI Literacy as a Key Driver of User Experience in AI-Powered Assessment: Insights from Socratic Mind

2025-07-29

AI Summary

This study investigates how AI literacy—and not merely prior technical exposure—influences undergraduate students’ perceptions of Socratic Mind, an AI-driven formative assessment tool. Grounded in Self-Determination Theory and established user experience frameworks, a survey was administered to 309 undergraduates, and data were analyzed using partial least squares structural equation modeling. Results demonstrate that AI literacy—particularly self-efficacy, conceptual understanding, and application competence—significantly and positively predicts perceived usability, user satisfaction, and engagement, which in turn enhance perceived learning outcomes. In contrast, prior technical exposure alone showed no significant effect. This study provides the first empirical evidence that AI literacy serves as a critical antecedent to user experience, superseding experiential familiarity with technology. Based on these findings, we propose adaptive support design principles tailored to varying levels of AI literacy, thereby advancing theoretically grounded, human-centered development of educational AI tools.

Technology Category

• Humans and AI: User Experience and Usability
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As Artificial Intelligence (AI) tools become increasingly embedded in higher education, understanding how students interact with these systems is essential to supporting effective learning. This study examines how students' AI literacy and prior exposure to AI technologies shape their perceptions of Socratic Mind, an interactive AI-powered formative assessment tool. Drawing on Self-Determination Theory and user experience research, we analyze relationships among AI literacy, perceived usability, satisfaction, engagement, and perceived learning effectiveness. Data from 309 undergraduates in Computer Science and Business courses were collected through validated surveys. Partial least squares structural equation modeling showed that AI literacy - especially self-efficacy, conceptual understanding, and application skills - significantly predicts usability, satisfaction, and engagement. Usability and satisfaction, in turn, strongly predict perceived learning effectiveness, while prior AI exposure showed no significant effect. These findings highlight that AI literacy, rather than exposure alone, shapes student experiences. Designers should integrate adaptive guidance and user-centered features to support diverse literacy levels, fostering inclusive, motivating, and effective AI-based learning environments.

Problem

• How AI literacy affects student experience in AI-powered assessment tools
• Relationship between AI literacy, usability, and learning effectiveness
• Impact of prior AI exposure versus AI literacy on student perceptions

Innovation

• AI literacy predicts usability and engagement
• User-centered design enhances learning effectiveness
• Adaptive guidance supports diverse literacy levels