🤖 AI Summary
This work addresses the challenge of efficiently training hundred-billion-parameter sparse Mixture-of-Experts (MoE) large language models on a single eight-GPU node, substantially reducing memory and optimizer state overhead. The authors propose a unified training framework that, for the first time, integrates reversible recurrent architectures, state-preserving progressive expansion, quantized base expert weights with low-rank adapters (TQP), Top-12 routing, and an 8K-token context window. Starting from a dense small model, this approach enables end-to-end evolution into a 120B-parameter MoE model with 5.93B activated parameters. The resulting model achieves a training loss of 1.78 and demonstrates strong multilingual (including Indic languages) and code capabilities. The model, tokenizer, and training code are publicly released.
📝 Abstract
This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end. LightningLM 0.1V is a recurrence-backbone language model family grown in four stages from a small dense seed, through a 5B and a 9B mixture of experts, to a 120B model with 460 routed experts under top-12 routing. Each larger model is grown from the trained weights of the smaller one; active parameters rise monotonically from 1.78B at the dense seed to 5.93B at 120B (about 5% of the 118.67B stored). The full lineage runs on single nodes, the larger stages at 8K context, reaching a released training loss of 1.78 at 120B scale.
This is a systems and experience report. It is organized around three disciplines. Reversibility: a reversible recurrence stack reconstructs activations in the backward pass instead of storing them, holding activation memory flat as the model grows. State-preserving growth: each expansion (dense to MoE, shallow to deep, few experts to many) is given as a reproducible principle paired with the failure that results from getting it wrong; several failures are silent. Single-node economics: the 120B trains through TQP, a strategy of quantized base expert weights and trained low-rank adapters that carries optimizer state on 2.26B adapter parameters rather than 100B+ resident in routed experts, cutting expert-path optimizer state by a factor of ~45.
What is new is the integration of known primitives, not any primitive in isolation: one grown lineage running end to end on a single node, documented at practitioner level, with per-domain held-out loss as evidence that targeted capabilities (multilingual Indic competence, code) were learned by construction. Model family, tokenizer, and training code are released.