GLiClass: Generalist Lightweight Model for Sequence Classification Tasks

2025-08-11

AI Summary

Existing sequence classification methods struggle to simultaneously achieve efficiency, accuracy, and zero-shot capability, while suffering from poor instruction alignment, scalability to large label sets, and inadequate modeling of complex semantics. To address these challenges, this paper proposes GLiClass—a lightweight, general-purpose model for sequence classification. Its core innovations include: (1) adapting the GLiNER architecture to sequence classification via prompt-based learning and a cross-encoder structure for efficient text-label matching; and (2) integrating Proximal Policy Optimization (PPO) to optimize multi-label classification objectives, enabling human feedback incorporation and robustness in data-scarce settings. Experiments demonstrate that GLiClass maintains embedding-level inference efficiency while substantially outperforming conventional generative models and cross-encoders in zero-shot and few-shot accuracy. Moreover, it exhibits superior generalization and instruction-following capability under dynamic label spaces and complex logical constraints.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Classification is one of the most widespread tasks in AI applications, serving often as the first step in filtering, sorting, and categorizing data. Since modern AI systems must handle large volumes of input data and early pipeline stages can propagate errors downstream, achieving high efficiency and accuracy is critical. Moreover, classification requirements can change dynamically based on user needs, necessitating models with strong zero-shot capabilities. While generative LLMs have become mainstream for zero-shot classification due to their versatility, they suffer from inconsistent instruction following and computational inefficiency. Cross-encoders, commonly used as rerankers in RAG pipelines, face a different bottleneck: they must process text-label pairs sequentially, significantly reducing efficiency with large label sets. Embedding-based approaches offer good efficiency but struggle with complex scenarios involving logical and semantic constraints. We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks. Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios. Additionally, we adapted proximal policy optimization (PPO) for multi-label text classification, enabling training classifiers in data-sparse conditions or from human feedback.

Problem

• Efficient and accurate sequence classification in AI pipelines
• Dynamic classification needs requiring strong zero-shot capabilities
• Handling large label sets and complex semantic constraints

Innovation

• Adapts GLiNER architecture for sequence classification
• Uses PPO for multi-label text classification
• Balances efficiency and zero-shot learning flexibility