🤖 AI Summary
This study addresses the challenge of efficiently and transparently aligning learning resources to structured competency frameworks. The authors propose an end-to-end alignment pipeline that first segments instructional and assessment content into granular fragments, then enhances candidate competency retrieval using a competency knowledge graph. A large language model (LLM) selects the most relevant competencies and generates traceable textual evidence, followed by graph-constrained inference to refine predictions. Integrating LLM reasoning, BM25 retrieval, and graph-based constraints, the method substantially outperforms zero-shot, few-shot, and traditional supervised baselines on the UTC Computer Science dataset, achieving a fragment-level micro F1 of 0.57, macro F1 of 0.50, resource-level macro F1 of 0.51, and MRR of 0.82, while producing auditable alignment evidence.
📝 Abstract
Linking learning resources to a structured competency framework is key to enabling competency-based search and curriculum analytics in Learning Management Systems (LMS). However, manual tagging is labor-intensive, and fully automatic methods often lack transparency. In this paper, we present an end-to-end alignment pipeline that uses a large language model (LLM) as a constrained, evidence-producing tagger. LMS resources -both instructional content and assessments -are first segmented into meaningful pedagogical fragments. For each fragment, a small set of candidate competencies is retrieved from structured competency profiles enriched with graph-based context. The LLM then selects the most relevant competencies from this set and provides supporting evidence spans from the fragment text. These predictions are refined using the structure of the competency graph and aggregated at the resource level. We evaluate our approach on a dataset built from the Computer Science department's competency referential at the Université de Technologie de Compiègne (UTC), covering 22 competencies across multiple course materials. Our LLM+BM25+Graph (LBG) pipeline achieves strong results, with a micro-F1 of 0.57 and macro-F1 of 0.50 at the fragment level, 0.51 macro-F1 at the resource level, and an MRR of 0.82outperforming zero-shot and few-shot LLM variants, retrieval/similarity baselines, and supervised classifiers -while also producing more mechanically traceable evidence spans to support human auditing and educational analysis.