Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning

2026-07-24

AI Summary

This work addresses the tendency of large language models to provide direct answers in educational settings, lacking Socratic guidance. To remedy this, the authors propose a two-stage alignment approach: first applying supervised fine-tuning for warm-up, followed by Group Relative Policy Optimization (GRPO) incorporating reward heuristics based on cognitive depth, curiosity engagement, and directness. The method transforms Qwen2.5-7B into a Socratic tutor. The study introduces the SocraticEdu dataset and novel evaluation metrics—Scaffolding Effectiveness (SE) and Conversation Depth (CD)—and surprisingly finds that removing the directness penalty enhances guided learning. On 30 held-out questions, the best GRPO variant improves SE from 30.0% to 63.3% and reduces keyword leakage to 13.3%, substantially outperforming the unaligned Qwen-72B baseline.

Abstract

Large language models (LLMs) deployed in educational settings often behave as direct answerers: they disclose target concepts in the opening turn instead of guiding students through progressive inquiry, as Socratic pedagogy prescribes. We present HeuristicEdu, a two-phase pipeline that aligns Qwen2.5-7B toward Socratic tutoring via supervised warm-up and Group Relative Policy Optimization (GRPO). Training uses SocraticEdu, 797 multi-turn Chinese children's science dialogues reconstructed from a live platform, with a heuristic reward over cognitive depth (R_cog), curiosity engagement (R_eng), and directness (R_dir), together with a K_query correction for student-introduced terms. We introduce Scaffolding Effectiveness (SE) and Conversation Depth (CD) to evaluate outcomes beyond surface fluency. On 30 held-out questions, the best GRPO variant improves SE from 30.0% to 63.3% and lowers keyword leakage from 30.0% to 13.3%. Notably, this best variant omits the directness penalty during optimization, suggesting that explicit anti-leakage terms can conflict with gradient-based behavioral alignment. An unaligned Qwen-72B baseline reaches 0% SE and 96.7% leakage, showing that scale alone does not induce Socratic behavior.

Problem

• Socratic tutoring
• educational LLMs
• direct answering
• student guidance
• cognitive depth

Innovation

• Socratic tutoring
• Heuristic Reinforcement Learning
• Group Relative Policy Optimization
• Scaffolding Effectiveness
• Reward Design