Evaluating Machine Expertise: How Graduate Students Develop Frameworks for Assessing GenAI Content

2025-04-24

AI Summary

This study investigates how graduate students evaluate the professional credibility of large language model (LLM)-generated content in Web environments, focusing on the mechanisms underlying their evaluation frameworks. Method: A mixed-methods approach was employed—including surveys, analysis of LLM interaction logs, and semi-structured in-depth interviews—combined with thematic coding and cross-case pattern analysis. Contribution/Results: The study identifies three interdependent factors—professional identity, verification competence, and system navigation experience—that jointly shape evaluative reasoning. Crucially, students do not uniformly accept or reject LLM outputs; instead, they enact discipline-specific “gatekeeping” behaviors—dynamically defending core professional boundaries across conceptual (management), creative (design), and technical (programming) tasks. These findings provide empirically grounded insights for designing AI-mediated Web platforms with domain-adapted evaluation support mechanisms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

This paper examines how graduate students develop frameworks for evaluating machine-generated expertise in web-based interactions with large language models (LLMs). Through a qualitative study combining surveys, LLM interaction transcripts, and in-depth interviews with 14 graduate students, we identify patterns in how these emerging professionals assess and engage with AI-generated content. Our findings reveal that students construct evaluation frameworks shaped by three main factors: professional identity, verification capabilities, and system navigation experience. Rather than uniformly accepting or rejecting LLM outputs, students protect domains central to their professional identities while delegating others--with managers preserving conceptual work, designers safeguarding creative processes, and programmers maintaining control over core technical expertise. These evaluation frameworks are further influenced by students' ability to verify different types of content and their experience navigating complex systems. This research contributes to web science by highlighting emerging human-genAI interaction patterns and suggesting how platforms might better support users in developing effective frameworks for evaluating machine-generated expertise signals in AI-mediated web environments.

Problem

• How graduate students evaluate machine-generated expertise in LLM interactions
• Factors shaping evaluation frameworks: identity, verification, system experience
• Balancing acceptance and delegation of AI outputs based on professional domains

Innovation

• Qualitative study combining surveys, transcripts, interviews
• Evaluation frameworks based on identity, verification, navigation
• Domain-specific delegation of LLM outputs by professionals