CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming

2026-07-06

AI Summary

Deploying large language models (LLMs) in K–12 education raises concerns regarding privacy, cost, and reliance on proprietary systems, while small language models (SLMs) lack tailored evaluation benchmarks—particularly in data-scarce domains like block-based programming. To address this gap, this work proposes CSTutorBench, the first context-sensitive, pedagogy-driven benchmark for evaluating SLMs in VEX VR block-based programming instruction. It comprises 17 scenario-based questions grounded in established teaching principles and employs a human-in-the-loop LLM-as-judge scoring mechanism. Evaluation of 11 models (ranging from 4B to 120B parameters) reveals that model family and instruction-tuning strategy are stronger predictors of tutoring quality than parameter count alone. While most models exhibit adequate surface-level linguistic competence, they often lack deeper pedagogical behaviors; however, integrating pedagogically informed prompt engineering significantly improves performance across ten of the evaluated models.

Abstract

Large language models are increasingly explored as AI tutors, yet deploying them in K-12 settings raises concerns around privacy, cost, and reliance on proprietary models. Small language models (SLMs) offer a promising alternative, but selecting the right model for a specific educational context remains difficult, particularly when the target domain, such as block-based programming, is largely absent from model training data. We introduce CSTutorBench, a benchmark for evaluating language models as CS tutors in VEX VR, a block-based robotics environment. The benchmark comprises 17 scenario-based questions scored against a pedagogical rubric grounded in established tutoring and feedback research, with a human-in-the-loop LLM-as-judge pipeline for evaluation. Preliminary findings across 11 models (4B-120B parameters) reveal that models perform well on surface-level criteria such as vocabulary and tone but struggle with deeper pedagogical behaviors, particularly avoiding answer leakage and engaging with student debugging histories. In our sample, model family and instruction-tuning approach appear to be better predictors of tutoring quality than parameter count alone, though the small number of models limits the strength of this conclusion. A targeted prompt revision grounded in recent educational prompt engineering research improved scores for 10 of 11 models. These results underscore the value of context-specific, pedagogically grounded benchmarks for SLM selection in educational deployment.

Problem

• small language models
• AI tutors
• block-based programming
• educational benchmarking
• K-12 education

Innovation

• small language models
• educational benchmarking
• block-based programming
• LLM-as-judge
• prompt engineering