question generation

Low credibility, poor scalability, and ethical risks—including bias, privacy violations, and lack of transparency—hinder the use of generative AI, particularly large language models (LLMs), for autonomously generating high-quality, context-aware survey questions in engineering education research. Method: We propose a methodology framework for generative AI in educational surveys, introducing the first Synthetic Question–Response Analysis (SQRA) model grounded in Activity Theory to systematically model mediating mechanisms of learner engagement and associated ethical risks. Validation employs dual pathways—AI-to-AI and AI-to-human—via sentiment analysis, lexical statistics, and structured text analysis. Contribution/Results: Empirical findings demonstrate that prompt engineering combined with multidimensional validation significantly enhances question appropriateness, reliability, and validity. The study delineates the effective operational boundaries of AI-generated survey items and establishes a reusable, auditable, and scalable paradigm for intelligent data collection in educational empirical research.

This study investigates whether existing probing methods genuinely capture large language models’ awareness of evaluation contexts or merely reflect superficial structural cues from prompt formats. For the first time, it systematically disentangles evaluation context from prompt format by constructing a controlled 2×2 dataset and applying diagnostic text rewrites, thereby assessing linear probe effectiveness under partially constrained prompt structures. The results demonstrate that probe signals primarily stem from structural features of benchmark formulations rather than semantic understanding: probe performance substantially degrades when free-form prompts are used. This finding reveals the confounding influence of structural artifacts in current research on model awareness, undermining the reliability of prior conclusions and highlighting the critical need to distinguish genuine semantic comprehension from spurious dependencies on prompt formatting.

This work addresses the scarcity of high-complexity evaluation benchmarks for Retrieval-Augmented Generation (RAG) systems in specialized domains, particularly those requiring multimodal inputs and multi-hop reasoning. To bridge this gap, the authors propose a multi-agent collaborative framework that integrates recursive context refinement, adversarial fact verification, and expert role simulation to automatically generate verified, domain-specific, multimodal multi-hop question-answering datasets. The approach constructs high-quality datasets across four professional domains, each featuring an average reasoning depth exceeding 2.3 hops. Experimental results demonstrate substantial improvements in RAG systems’ reasoning capabilities, retrieval accuracy, and factual consistency when handling complex documents, thereby validating the framework’s novelty, effectiveness, and scalability.

This study investigates performance bottlenecks of generative AI in multimodal STEM education assessment, specifically why large language models (LLMs) significantly underperform humans on visually grounded questions. Method: We construct a novel, fine-grained dataset of 201 university-level multimodal STEM items—annotated for image type, role, visual complexity, and format—and conduct human–AI comparative experiments (n = 546 students) across four LLM families and five prompting strategies. Contribution/Results: We establish that human performance is primarily modulated by disciplinary domain, whereas AI accuracy is strongly contingent on visual features and item structural design. The best-performing ensemble method (majority voting) achieves only 58.5% accuracy—significantly below human performance across all image-containing item types. Based on these findings, we propose the “Feature Discriminability” framework: a principled, actionable paradigm for item design that enhances detectability of AI-generated responses and strengthens academic integrity in multimodal assessment.

Current Document VQA evaluation methods overlook semantic plausibility and multimodal grounding (i.e., textual semantics + visual grounding), treating hallucinations and factual errors equivalently—thus failing to reflect models’ true reasoning capabilities. To address this, we propose the first configurable evaluation framework that jointly models semantic consistency and visual spatial localization. Our method integrates OCR text alignment, visual region localization, and joint semantic consistency scoring, with user-controllable weighting parameters validated via human judgments. Experiments demonstrate that our framework significantly improves evaluation discriminability: it achieves human-preferred re-ranking on major benchmarks and effectively identifies hallucinations and reasoning failures. By enabling principled calibration and robust multimodal assessment, our approach establishes a more reliable, interpretable, and human-aligned benchmark for Document VQA evaluation.

This work addresses the vulnerability of self-generated question-answering (QA) as a training signal for language models, which is prone to selection bias and excessive compliance with embedded instructions. The study systematically reveals the model’s preferential selection of document evidence and its high adherence to implicit directives during self-generated QA. To mitigate these issues without altering the training pipeline, the authors propose lightweight interventions: fixing the questioning objective and filtering instruction-laden segments. Through comprehensive empirical evaluations—including multi-model comparisons, prompt diversity analysis, text coverage assessment, and instruction injection detection—the approach reduces average instruction compliance from 88% to 13% while preserving nearly all clean textual content. This significantly enhances the reliability and robustness of self-generated QA data for model training.

This study addresses whether the outputs of small language models (SLMs) in psychometric tasks stem from genuine semantic reasoning or are primarily driven by artifacts of prompt formulation. The authors propose the first diagnostic framework capable of disentangling the influence of such prompt artifacts, systematically manipulating role framing, instructions, item content, and option labels while employing controlled experiments and variance decomposition techniques to quantify the relative contributions of semantic signals versus prompt-induced artifacts. Findings reveal that prompt artifacts frequently dominate model responses, substantially undermining their psychometric validity. The proposed framework not only effectively identifies these confounding influences but also offers a novel pathway for evaluating and enhancing the semantic comprehension capabilities of large language models.

This work addresses the lack of procedural reasoning benchmarks in current AI learning systems that simultaneously support natural language expression, knowledge traceability, and multi-hop reasoning. The authors propose a question generation and validation framework grounded in the Task–Method–Knowledge (TMK) model, which systematically constructs high-quality question-answer pairs through TMK-guided generation, transcription-based textual constraints, posterior filtering, and verification via closed evidence units. This approach achieves an effective balance between natural language fluency and structured knowledge representation. Evaluated across 23 instructional topics yielding 690 question-answer pairs, the method demonstrates strong performance under strict TMK criteria, attaining a 96.5% knowledge grounding rate and a 92.6% usability rate—significantly outperforming baseline approaches.

This work addresses the challenge of hidden misconceptions in student problem-solving, where learners often arrive at correct answers through flawed reasoning—rendering such errors invisible to conventional automated feedback systems. To tackle this issue, the authors propose a hierarchical assessment framework that decouples answer correctness from the validity of the solution process. They design a “detect–verify–escalate” pipeline that, under conditions of high uncertainty, triggers diagnostic follow-up questions rather than immediate alerts. Integrating a fine-tuned classifier with an open-source reasoning model and grounding the approach in educational theory, the system achieves low false-positive rates while maintaining high coverage of misconception detection. Empirical results demonstrate that the framework identifies 84% of hidden misconceptions, substantially reducing teacher intervention load and supporting dual deployment modes—teacher review and autonomous tutoring.

This work addresses the lack of verifiable mechanisms in existing retrieval-augmented systems for assessing the true influence of cited sources on generated answers. The authors propose a seven-stage framework that establishes three measurable layers of transparency in multi-hop question answering: answer correctness, citation fidelity, and single-document influence. For the first time, they integrate causal mediation analysis with database provenance theory to formally characterize the “citation-influence gap” and define faithfulness conditions via token-level KL divergence. Leveraging techniques such as citation-aware generation, attribution auditing, and leave-one-out interventions, the system achieves 53.53% answer accuracy and 71.55% average citation fidelity on the HotpotQA validation set, revealing a frequent misalignment between cited references and their actual evidential impact.