Constructing a Question-Answering Simulator through the Distillation of LLMs

2025-09-11

AI Summary

To address the challenge of secure offline training for Educational Recommendation Systems (ERS) in the absence of authentic student feedback, this paper proposes LDSim—a lightweight question-answering simulator that distills knowledge and multi-step reasoning capabilities from Large Language Models (LLMs). LDSim integrates knowledge distillation, sequential student behavior modeling, and LLM output alignment to predict student response correctness solely from historical Q&A interactions, enabling high-fidelity learning behavior simulation without real-time user engagement. As the first work to jointly distill both domain-specific knowledge and stepwise reasoning skills from LLMs into a compact model, LDSim achieves state-of-the-art performance on both question-answer simulation and knowledge tracing tasks. It delivers millisecond-scale inference latency while maintaining high prediction accuracy, thereby establishing a robust foundation for safe, offline ERS training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The question-answering (QA) simulator is a model that mimics real student learning behaviors and predicts their correctness of their responses to questions. QA simulators enable educational recommender systems (ERS) to collect large amounts of training data without interacting with real students, thereby preventing harmful recommendations made by an undertrained ERS from undermining actual student learning. Given the QA history, there are two categories of solutions to predict the correctness, conducting the simulation: (1) LLM-free methods, which apply a traditional sequential model to transfer the QA history into a vector representation first, and make predictions based on the representation; (2) LLM-based methods, which leverage the domain knowledge and reasoning capability of LLM to enhence the prediction. LLM-free methods offer fast inference but generally yield suboptimal performance. In contrast, most LLM-based methods achieve better results, but at the cost of slower inference speed and higher GPU memory consumption. In this paper, we propose a method named LLM Distillation based Simulator (LDSim), which distills domain knowledge and reasoning capability from an LLM to better assist prediction, thereby improving simulation performance. Extensive experiments demonstrate that our LDSim achieves strong results on both the simulation task and the knowledge tracing (KT) task. Our code is publicly available at https://anonymous.4open.science/r/LDSim-05A9.

Problem

• Mimics student learning behaviors for QA
• Predicts student response correctness efficiently
• Distills LLM knowledge to improve simulation

Innovation

• Distills LLM knowledge for simulation
• Enhances prediction with reasoning capability
• Improves performance on educational tasks