SRJudge: Empowering Large Language Models with Selective Reasoning for Fine-Grained Knowledge Concept Tagging

2026-09-29

AI Summary

This study addresses the challenge of precisely annotating knowledge concepts from large-scale candidate sets, where the prohibitively high dimensionality of the decision space hinders the performance of large language models (LLMs). To this end, we propose SRJudge, a novel framework that introduces a three-stage "Screening-Reasoning-Judging" architecture. Specifically, the method first employs a fine-tuned BERT model for candidate pruning, followed by a lightweight LLM to execute reasoning. It further achieves fine-grained alignment by integrating dynamic reward reinforcement learning with an LLM-based judging mechanism. We validate the proposed approach on newly constructed biology and physics datasets. Experimental results demonstrate that SRJudge significantly outperforms existing state-of-the-art baselines, confirming its effectiveness in navigating high-dimensional decision spaces for accurate knowledge concept annotation.

Abstract

Knowledge concept tagging aims to assign specific concept or topic labels to educational content, which is essential for both educators and learners in traditional and online teaching practices. Recent work has explored large language models (LLMs) for this task, achieving promising performance. However, LLMs still struggle to select the correct concept from a large-scale candidate set due to the high dimensionality of the decision space. In this paper, we propose a novel three-stage Select-Reason-Judge (SRJudge) framework, which empowers LLMs with selective reasoning capability for fine-grained knowledge concept tagging. Specifically, the Selector in Stage 1 first narrows the candidate concepts to a top-K shortlist by fine-tuning a small language model (SLM), e.g., BERT, since the top-$K$ predictions hit the correct concept in most cases, thereby reducing the decision space of correct candidates. Next, the Stage 2 Reasoner employs a lightweight LLM for refined reasoning over the shortlisted candidates. It further integrates an improved reinforcement learning strategy with a dynamic task-specific reward function and a pruning mechanism to better align with human reasoning preferences. Finally, a larger LLM acts as a judger that evaluates the overall rationality of the reasoning process and its explanations to determine the final output. In addition, we construct two high-quality datasets for further validation, i.e., the biology dataset S_Bio and the physics dataset S_Phy. Experimental results demonstrate that our method consistently outperforms state-of-the-art baselines across benchmark datasets, verifying its effectiveness and superiority. Resources are available at: https://github.com/Nicozwy/SRJudge.

Problem

• Knowledge Concept Tagging
• Large Language Models
• Fine-Grained Labeling
• Decision Space

Innovation

• Selective Reasoning
• Knowledge Concept Tagging
• Large Language Models
• Reinforcement Learning
• SRJudge Framework