HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

2026-05-11

AI Summary

This study addresses the current lack of efficient, open-source large language models tailored for Hebrew that support long-context processing. The work proposes the first native Hebrew-specific Mixture-of-Experts (MoE) model based on NVIDIA’s Nemotron-3 sparse architecture, capable of handling context lengths up to 65,536 tokens. The approach integrates a three-stage curriculum learning strategy progressing from easy to hard tasks, a continual anti-forgetting anchoring mechanism, and supervised fine-tuning on a dataset of two million Hebrew–English bilingual examples. Despite activating only approximately 3 billion parameters during inference, the model achieves a throughput nine times higher than comparable models and attains a Hebrew reasoning accuracy of 73.8%, substantially outperforming DictaLM-3.0-24B-Thinking.

Abstract

We present Hebatron, a Hebrew-specialized open-weight large language model built on the NVIDIA Nemotron-3 sparse Mixture-of-Experts architecture. Training employs a three-phase easy-to-hard curriculum with continuous anti-forgetting anchoring, followed by supervised fine-tuning on 2 million bilingual Hebrew--English samples. The curriculum ordering alone yields a 3-point aggregate benchmark gain over the reversed configuration. Hebatron achieves a Hebrew reasoning average of 73.8\%, outperforming DictaLM-3.0-24B-Thinking (68.9\%) and remaining competitive with Gemma-3-27B-IT on GSM8K-HE and Israeli Trivia, while activating only 3B parameters per forward pass across a 30B-parameter model, delivering approximately 9 times higher inference throughput at native context lengths up to 65,536 tokens. To our knowledge, this is the first language-specific adaptation of the Nemotron-3 architecture for any target language, and the first open-weight Hebrew-specialized MoE model with native long-context support. Model weights are released openly to support further research in Hebrew and Semitic-language NLP.

Problem

• Hebrew NLP
• Mixture-of-Experts
• open-weight language model
• long-context modeling
• language-specific adaptation

Innovation

• Mixture-of-Experts
• Hebrew-specialized LLM
• curriculum learning
• long-context modeling
• open-weight model