RigoChat 2: an adapted language model to Spanish using a bounded dataset and reduced hardware

2025-03-11

AI Summary

To address the high computational cost and deployment challenges of large language models (LLMs) on resource-constrained devices for Spanish-language tasks, this paper proposes a lightweight adaptation framework. The method builds upon a small pre-trained model and integrates parameter-efficient fine-tuning (PEFT), instruction tuning, Spanish-domain data distillation, and quantized inference. It is the first work to achieve Spanish-specific optimization of state-of-the-art (SOTA) LLMs under extreme constraints: minimal labeled data and a single 16GB GPU. Experimental results demonstrate substantial improvements over baselines in both understanding and generation tasks, with 87% reduction in training time and 76% lower GPU memory consumption. The approach achieves a favorable trade-off among performance, efficiency, and deployability, establishing a reproducible, low-cost paradigm for multilingual LLM adaptation in resource-limited settings.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsSearch and Optimization: Learning to Search

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Large Language Models (LLMs) have become a key element of modern artificial intelligence, demonstrating the ability to address a wide range of language processing tasks at unprecedented levels of accuracy without the need of collecting problem-specific data. However, these versatile models face a significant challenge: both their training and inference processes require substantial computational resources, time, and memory. Consequently, optimizing this kind of models to minimize these requirements is crucial. In this article, we demonstrate that, with minimal resources and in a remarkably short time, it is possible to enhance a state-of-the-art model, specifically for a given language task, without compromising its overall capabilities using a relatively small pretrained LLM as a basis. Specifically, we present our use case, RigoChat 2, illustrating how LLMs can be adapted to achieve superior results in Spanish-language tasks.

Problem

• Optimize LLMs for reduced computational resources
• Adapt LLMs for specific language tasks efficiently
• Enhance Spanish-language performance with minimal resources

Innovation

• Adapted LLM for Spanish using minimal resources
• Enhanced model with small pretrained LLM basis
• Optimized for reduced hardware and time requirements