Methodological Foundations for AI-Driven Survey Question Generation

2025-05-02

AI Summary

Low credibility, poor scalability, and ethical risks—including bias, privacy violations, and lack of transparency—hinder the use of generative AI, particularly large language models (LLMs), for autonomously generating high-quality, context-aware survey questions in engineering education research. Method: We propose a methodology framework for generative AI in educational surveys, introducing the first Synthetic Question–Response Analysis (SQRA) model grounded in Activity Theory to systematically model mediating mechanisms of learner engagement and associated ethical risks. Validation employs dual pathways—AI-to-AI and AI-to-human—via sentiment analysis, lexical statistics, and structured text analysis. Contribution/Results: Empirical findings demonstrate that prompt engineering combined with multidimensional validation significantly enhances question appropriateness, reliability, and validity. The study delineates the effective operational boundaries of AI-generated survey items and establishes a reusable, auditable, and scalable paradigm for intelligent data collection in educational empirical research.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

This paper presents a methodological framework for using generative AI in educational survey research. We explore how Large Language Models (LLMs) can generate adaptive, context-aware survey questions and introduce the Synthetic Question-Response Analysis (SQRA) framework, which enables iterative testing and refinement of AI-generated prompts prior to deployment with human participants. Guided by Activity Theory, we analyze how AI tools mediate participant engagement and learning, and we examine ethical issues such as bias, privacy, and transparency. Through sentiment, lexical, and structural analyses of both AI-to-AI and AI-to-human survey interactions, we evaluate the alignment and effectiveness of these questions. Our findings highlight the promise and limitations of AI-driven survey instruments, emphasizing the need for robust prompt engineering and validation to support trustworthy, scalable, and contextually relevant data collection in engineering education.

Problem

• Developing AI methods for adaptive educational survey question generation
• Evaluating AI-generated questions via testing frameworks and ethical considerations
• Assessing alignment and effectiveness of AI-driven survey instruments

Innovation

• LLMs generate adaptive, context-aware survey questions
• SQRA framework enables iterative AI prompt refinement
• Activity Theory guides AI-mediated engagement analysis