Small Language Models in the Real World: Insights from Industrial Text Classification

2025-05-21

AI Summary

Lightweight decoder-only small language models (≤7B parameters) face critical bottlenecks—low inference efficiency, high prompt sensitivity, and excessive GPU memory consumption—when deployed for industrial text classification tasks (e.g., emails, legal documents, and ultra-long academic texts). Method: We propose a VRAM-aware evaluation framework integrating dynamic prompt template design, task-adapted supervised fine-tuning, and joint memory-throughput benchmarking. Contribution/Results: This work presents the first cross-task, long-text industrial evaluation of small models’ inference efficiency and classification robustness. It reveals a nonlinear trade-off between prompt quality and model scale. Experiments across three real-world tasks achieve >92% accuracy, with 60% GPU memory reduction and 45% latency improvement over generic prompting—demonstrating strong viability for on-device or edge-deployed lightweight NLP systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With the emergence of ChatGPT, Transformer models have significantly advanced text classification and related tasks. Decoder-only models such as Llama exhibit strong performance and flexibility, yet they suffer from inefficiency on inference due to token-by-token generation, and their effectiveness in text classification tasks heavily depends on prompt quality. Moreover, their substantial GPU resource requirements often limit widespread adoption. Thus, the question of whether smaller language models are capable of effectively handling text classification tasks emerges as a topic of significant interest. However, the selection of appropriate models and methodologies remains largely underexplored. In this paper, we conduct a comprehensive evaluation of prompt engineering and supervised fine-tuning methods for transformer-based text classification. Specifically, we focus on practical industrial scenarios, including email classification, legal document categorization, and the classification of extremely long academic texts. We examine the strengths and limitations of smaller models, with particular attention to both their performance and their efficiency in Video Random-Access Memory (VRAM) utilization, thereby providing valuable insights for the local deployment and application of compact models in industrial settings.

Problem

• Evaluating small language models for industrial text classification tasks
• Assessing efficiency and VRAM usage of compact models in real-world applications
• Exploring prompt engineering and fine-tuning methods for transformer-based classification

Innovation

• Evaluates prompt engineering and fine-tuning methods
• Focuses on small models for industrial text classification
• Analyzes VRAM efficiency for local deployment