GTA: Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models

2025-09-15

AI Summary

Pure reinforcement learning (RL) suffers from slow convergence and inefficient exploration in NLP tasks, while supervised fine-tuning (SFT) faces inherent performance ceilings and lacks rigorous theoretical foundations. Method: This paper proposes the Guess-Think-Answer (GTA) unified training framework, which models text classification as a three-stage reasoning process—“guess → reflect → answer”—and jointly optimizes SFT’s cross-entropy loss and RL’s reward signal within a single paradigm. To mitigate objective conflict, GTA introduces loss masking and gradient constraint mechanisms, accompanied by theoretical convergence analysis. Contribution/Results: Experiments on four text classification benchmarks demonstrate that GTA significantly outperforms both pure SFT and pure RL baselines, achieving faster convergence and higher performance ceilings.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Reinforcement Learning
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In natural language processing tasks, pure reinforcement learning (RL) fine-tuning methods often suffer from inefficient exploration and slow convergence; while supervised fine-tuning (SFT) methods, although efficient in training, have limited performance ceiling and less solid theoretical foundation compared to RL. To address efficiency-capability trade-off, we propose the Guess-Think-Answer (GTA) framework that combines the efficiency of SFT with the capability gains of RL in a unified training paradigm. GTA works by having the model first produce a provisional guess (optimized via cross-entropy loss), then reflect on this guess before generating the final answer, with RL rewards shaping both the final output and the format of the entire GTA structure. This hybrid approach achieves both faster convergence than pure RL and higher performance ceiling than pure SFT. To mitigate gradient conflicts between the two training signals, we employ loss masking and gradient constraints. Empirical results on four text classification benchmarks demonstrate that GTA substantially accelerates convergence while outperforming both standalone SFT and RL baselines.

Problem

• Combining supervised and reinforcement learning for text classification
• Addressing efficiency-capability trade-off in LLM fine-tuning
• Mitigating gradient conflicts between different training signals

Innovation

• Combines supervised and reinforcement learning in unified framework
• Uses Guess-Think-Answer structure with RL reward shaping
• Employs loss masking and gradient constraints for training