Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models

2026-07-30

AI Summary

Can the generic “helpfulness” rating of large language models (LLMs) effectively distinguish between directly providing answers and employing instructional scaffolding? This study addresses this question through a preregistered experiment comparing conversational and pedagogical strategies across three tutoring models within an identical student simulator. Using Claude Opus 4.8 and GPT-5.6 Sol as evaluators, alongside a determinism detector to quantify answer leakage and student independence, the findings reveal that while pedagogical strategies are clearly differentiated under teaching-specific metrics, they show no separation under generic helpfulness scores. Moreover, answer leakage consistently undermines student independent reasoning, and helpfulness rankings exhibit inconsistency across evaluator models. The work advocates for combining domain-specific teaching evaluations with process-oriented indicators to reliably assess tutoring quality.

Abstract

LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance? We audit this signal in a pre-registered study. Within each of three tutor bases, we compare conversational and pedagogical policies instantiated with the same underlying model and paired with one fixed weak simulated student. Deterministic detectors measure answer leakage and next-turn independent work. Claude Opus 4.8 is the frozen, condition-blind primary judge. After the Opus scores were fixed, GPT-5.6 Sol was prospectively specified for a post hoc robustness audit of the same 1,179 confirmatory answer-phase tutor turns under the frozen helpfulness and pedagogy rubrics. On the primary base under Opus, the policies do not differ significantly in helpfulness but are perfectly rank-separated under the pedagogy rubric (Cliff's $|δ|{=}0.10$ vs. $1.0$). Across the two judges, pedagogy contrasts retain their direction where detected, whereas the helpfulness ordering is judge-contingent, reversing between judges on two of three bases. In an Opus-only ablation, seven primary-base policies span $2.3$ points in mean judged pedagogy within a $0.25$-point band of mean judged helpfulness. Separately, answer-revealing turns are followed by less independent student work on every base, a result that is judge-invariant by construction. In this controlled setting, general-purpose helpfulness is not a reliable pedagogy signal. Tutor evaluation should pair pedagogy-targeted rubrics with deterministic process measures.

Problem

• LLM tutoring
• helpfulness
• pedagogy
• evaluation rubric
• answer leakage

Innovation

• pedagogy signal
• LLM-judged helpfulness
• pre-registered audit
• deterministic process measures
• tutor evaluation