BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

2026-08-05

AI Summary

This work addresses the challenge of deploying large pre-trained language models for low-resource languages like Bengali, which face prohibitive computational costs and carbon footprints that hinder deployment on resource-constrained devices. The study proposes BnBERT-iPET, a lightweight few-shot language model that integrates the Lottery Ticket Hypothesis-based pruning with iterative pattern-exploiting training (iPET), retaining only 10% of the original BERT parameters. Evaluated on standard Bengali benchmarks, the model achieves up to 90% sparsity while matching the performance of state-of-the-art models such as Bangla Electra, Indic-BERT, and XLM-RoBERTa. This approach significantly reduces computational requirements without sacrificing accuracy, offering a novel and sustainable paradigm for green natural language processing in low-resource linguistic settings.

Abstract

Deep neural networks have shown impressive success in NLP tasks owing to their complex structure and huge number of edges. Achieving state-of-the-art performance in natural language processing with a large pre-trained model such as BERT is expensive and time-consuming, carries a large carbon footprint, and is difficult to realize on machines with minimal computational capability. This creates a barrier to training complex models for resource-constrained languages such as Bengali. However, in a complex neural model, not all edges are equally impactful, and the contributions of some of them can be neglected. Pruning promises to reduce the memory footprint of regular networks, shorten the training time of ever-growing networks, and increase inference efficiency without sacrificing comparable performance. In this work, we introduce BnBERT-iPET, a sparse few-shot language modeling approach for Bengali, and experimentally show that a lightweight few-shot-learned language model retaining only 10% of the edges of an initial model such as BERT can perform neck and neck with much larger models on challenging tasks for a resource-constrained language such as Bengali. By learning from few shots through iterative pattern exploiting training and achieving 90% sparsity with the Lottery Ticket Hypothesis pruning technique, our pruned BnBERT-iPET model proves to be a tough competitor to state-of-the-art language models such as Bangla Electra, Indic-BERT, and XLM-RoBERTa on downstream tasks over standard benchmark datasets of the Bengali language.

Problem

• Few-Shot Learning
• Model Pruning
• Resource-Constrained Languages
• Sparse Language Modeling
• Bengali NLP

Innovation

• sparse modeling
• few-shot learning
• Lottery Ticket Hypothesis
• BERT pruning
• Bengali NLP