assess competency depth

This study addresses the lack of reliable methods for evaluating how undergraduate computer science curricula align with international teaching guidelines such as CS2013 and CS2023, and how this alignment evolves across guideline revisions. The authors propose a human-in-the-loop analytical pipeline that structures course content and guideline knowledge units, employs semantic retrieval—incorporating reciprocal rank fusion and lightweight sentence embedding models—to generate matching candidates, and applies clearly defined coverage criteria for human validation, supplemented by Cohen’s kappa to assess inter-rater consistency. For the first time, the approach enables longitudinal measurement of curriculum alignment across three dimensions: topic coverage, competency expression, and cognitive depth, distinguishing structural gaps from changes due to updated standards. Empirical results reveal knowledge unit coverage rates of 50.9% for CS2013 and 49.7% for CS2023, competency coverage around 88%, but a notable decline in adherence to recommended cognitive depth—from 95% to 76%.

This study investigates the extent to which higher education curricula embed 21st-century core competencies and align with societal demands. To this end, we construct a dataset comprising 7,600 human-annotated samples and introduce a "Curricular Chain-of-Thought" (Curricular CoT) prompting strategy to enhance large language models’ reasoning capabilities in educational contexts, mitigate keyword-matching biases, and improve detection of subtle pedagogical evidence in lengthy texts. Experimental results demonstrate that detailed descriptions of teaching activities are the most informative; open-source models perform comparably to closed-source counterparts on coarse-grained competency mapping tasks; and while Curricular CoT yields a modest yet statistically significant performance gain, models still fall substantially short of human-level proficiency in fine-grained educational reasoning.

Intelligent tutoring systems (ITS) in curriculum-based online learning risk exacerbating academic achievement gaps among students. Method: This paper proposes CTGraph, the first self-supervised graph representation learning framework that explicitly incorporates curriculum structure priors to construct student behavior graphs—modeling multidimensional learning signals including learning pathways, content coverage, engagement intensity, and conceptual mastery. A graph neural network performs graph-level encoding to enable cross-cohort behavioral comparison and stage-wise difficulty localization. Contribution/Results: Experiments demonstrate that CTGraph accurately identifies at-risk students, pinpoints optimal intervention timing with fine-grained temporal resolution, and localizes specific knowledge gaps. It significantly enhances personalized instructional support while providing interpretable, pedagogically grounded insights—offering a transparent, equity-oriented technical pathway for adaptive education.

This study addresses the limitations of existing visualization literacy assessments, which predominantly rely on multiple-choice questions and struggle to measure higher-order competencies due to ceiling effects. To overcome this, the work introduces two web-based qualitative assessment methods—visualization critique and sketching tasks—implemented through online think-aloud protocols and data-driven drawing exercises, respectively. These approaches holistically capture users’ advanced literacy in interpreting, evaluating, and constructing visualizations. Leveraging interactive tools, controlled comparative experiments, and validity evidence from established scales such as CALVI and Mini-VLAT, the proposed methods significantly differentiate individuals across varying expertise levels—including researchers, students, and crowdworkers—thereby surpassing the constraints of traditional item formats and uncovering critical competency dimensions previously unmeasurable by existing instruments.

This study addresses the challenge of interpreting the relationship between programming assignment difficulty and student performance, a gap exacerbated by the limited interpretability of existing predictive models that hinders effective pedagogical refinement. To bridge this gap, the authors propose an interpretable analytical framework grounded in knowledge components (KCs), which can be either expert-defined or automatically extracted using large language models (LLMs). By quantifying the number of KCs involved in each assignment and measuring the shift in KCs between consecutive tasks, the framework elucidates how assignment design influences learning outcomes. Empirical evaluation across three introductory programming course datasets reveals that assignments involving a greater number of KCs correlate with poorer student performance, and abrupt KC transitions are significantly associated with learning disruptions. These findings enable the identification of poorly designed assignments, offering actionable insights for instructional diagnosis and improvement.

This study addresses a critical limitation in current learning analytics tools, wherein frequency-oriented visualizations often obscure rare yet educationally significant student feedback. To bridge the gap between quantitative visualization and qualitative educational research, the authors engaged STEM education researchers in analyzing student logs using the WordStream platform. Through an integrated approach combining thematic analysis, member checking, and mixed-methods user research, the study uncovered epistemological tensions educators face when repurposing quantitative codings for qualitative inquiry. Three core themes emerged: tool experience, disciplinary contextualization, and the integration of quantitative and qualitative paradigms. Building on these insights, the work proposes design principles for visualizations that support deep qualitative exploration, offering both theoretical grounding and practical guidance for the next generation of learning analytics tools.

This study addresses the challenge of accurately tracking students’ dynamic mastery of specific skills when the Q-matrix is unknown. Building upon dynamic cognitive diagnosis models, it compares a joint estimation approach—simultaneously inferring the Q-matrix and learning trajectories—with a two-step strategy that first estimates the Q-matrix and then analyzes skill development. Leveraging reading game data and item text embeddings, the research investigates vocabulary and comprehension growth among second- to third-grade students. The authors propose a bias-corrected two-step method and use simulation studies to delineate the conditions under which each approach performs best: joint modeling proves more reliable when the Q-matrix is uncertain and items vary across grade levels. Empirical results indicate that both methods identify a general trend toward mastering both skills, yet they diverge in estimating the proportion of partial mastery in third grade, underscoring the substantive impact of modeling choices on diagnostic conclusions.

Current evaluation methods struggle to finely assess how large language model agents actually utilize reusable skills and why they fail. This work proposes "skill coverage" as a test adequacy metric tailored to agent skills, translating natural language skill instructions into semi-structured behavioral constraints and evaluating whether these constraints are covered and successfully executed based on execution traces. This approach decouples skill usage from task outcomes, enabling actionable failure attribution. Experiments on SkillsBench reveal that existing agents cover only 38.66%–45.51% of skill constraints; further, reinforcing skills based on failed constraints yields an average task recovery rate of 16.0% across previously failed tasks.