🤖 AI Summary
Current large language models struggle to reliably generate diverse multiple-choice questions aligned with specific cognitive levels—such as comprehension, reasoning, and main idea identification. To address this limitation, this work proposes the ReQUESTA framework, which introduces a novel hybrid multi-agent architecture that decomposes question generation into distinct phases: planning, constrained generation, iterative evaluation, and post-processing. By integrating large language models with rule-based engines and cognitive taxonomies, ReQUESTA enables structured and controllable generation of high-quality items. Experimental results demonstrate that the generated questions exhibit superior psychometric properties, including higher difficulty and discrimination indices, and show strong alignment with reading comprehension ability. Expert evaluations further confirm significant improvements over baseline methods in thematic relevance, distractor coherence, and semantic plausibility.
📝 Abstract
Recent advances in large language models (LLMs) have made automated multiple-choice question (MCQ) generation increasingly feasible; however, reliably producing items that satisfy controlled cognitive demands remains a challenge. To address this gap, we introduce ReQUESTA, a hybrid, multi-agent framework for generating cognitively diverse MCQs that systematically target text-based, inferential, and main idea comprehension. ReQUESTA decomposes MCQ authoring into specialized subtasks and coordinates LLM-powered agents with rule-based components to support planning, controlled generation, iterative evaluation, and post-processing. We evaluated the framework in a large-scale reading comprehension study using academic expository texts, comparing ReQUESTA-generated MCQs with those produced by a single-pass GPT-5 zero-shot baseline. Psychometric analyses of learner responses assessed item difficulty and discrimination, while expert raters evaluated question quality across multiple dimensions, including topic relevance and distractor quality. Results showed that ReQUESTA-generated items were consistently more challenging, more discriminative, and more strongly aligned with overall reading comprehension performance. Expert evaluations further indicated stronger alignment with central concepts and superior distractor linguistic consistency and semantic plausibility, particularly for inferential questions. These findings demonstrate that hybrid, agentic orchestration can systematically improve the reliability and controllability of LLM-based generation, highlighting workflow design as a key lever for structured artifact generation beyond single-pass prompting.