EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

2026-07-21

AI Summary

This study addresses the lack of effective code embedding methods for visual programming languages such as Scratch by systematically evaluating four large language models combined with five embedding strategies on textual representations of Scratch programs. The authors construct datasets for token prediction and program functionality classification tasks, demonstrating that embedding models trained on large-scale Scratch data effectively integrate structural and semantic information. Notably, these embeddings accurately predict the functional correctness of student programs without requiring task-specific fine-tuning. The findings offer a transferable and efficient embedding framework for classroom-level learning analytics, thereby filling a critical gap in representation learning for visual programming education.

Abstract

The trend of embedding source code for machine learning applications also enables new opportunities in learning analytics in programming education, but which code embedding approach is most suitable for learning analytics remains an open question. A common approach to embedding source code lies in treating the code as a token sequence similar to natural language when training large language models~(LLMs). However, in case of visual block-based programming languages like Scratch, this approach cannot be applied directly. While text-based representations of block-based code can be created to apply LLMs to this problem, other dedicated embedding models could potentially exhibit improved performance by capturing additional structural information. In this paper, we therefore instantiate four LLMs and five different popular embedding approaches for Scratch programs, create a token-prediction and two different classification tasks with corresponding datasets, and empirically evaluate the models on them. Our experiments demonstrate that a transfer of code embeddings to the educational environment of Scratch is feasible. The embedding models trained on large open Scratch datasets capture relevant structural and semantic information about the code to enable learning analytics like predicting functional correctness of student programs, in the typically small classroom setting without requiring further task-specific model fine-tuning.

Problem

• code embeddings
• Scratch
• learning analytics
• block-based programming
• embedding evaluation

Innovation

• code embeddings
• Scratch
• block-based programming
• learning analytics
• transfer learning