π€ AI Summary
This work addresses the challenge of deploying large language models in production settings, where they often fail to meet low-latency requirements, while smaller models typically suffer from limited reasoning capabilities, hallucinations, and insufficient long-context memory. To overcome these limitations, the authors propose supervised fine-tuning small models such as Mistral on domain-specific natural languageβcode paired data, thereby internalizing domain knowledge directly into model weights and substantially reducing reliance on runtime context. Experimental results demonstrate that the fine-tuned small models outperform larger counterparts in code generation quality while maintaining lower latency. Load testing and real-world deployment confirm their efficiency and stability. Furthermore, the approach supports additional customer-specific fine-tuning without compromising general-purpose capabilities, offering a practical pathway toward efficient and accurate domain-specific code generation.
π Abstract
Many applications today use large language models for code generation; however, production systems have strict latency requirements that can be difficult to meet with large models. Small language models with a few billion parameters are resource efficient but may suffer from limited reasoning, hallucinations, or poor retention of longer context. Fine tuning improves task specific accuracy by embedding domain knowledge directly into model weights, reducing reliance on runtime context. We previously implemented a baseline natural language to code generation approach using a retrieval augmented generation pipeline that dynamically selected few shot examples to embed domain specific language context for a large language model. In this study, we evaluate small language models for generating domain specific language from natural language by fine tuning variants of Mistral and other models on a dataset of natural language code pairs. Our results show that the fine-tuned models achieve improved performance and latency on test datasets compared to larger models. We also demonstrate that the trained model can be further fine-tuned for customer specific scenarios without degrading general performance, helping resolve production issues. Load testing followed by production deployment confirmed optimal performance in terms of latency and quality. These findings demonstrate that task specific fine tuning with small language models provides an efficient, faster, and cost-effective alternative to large language models for domain specific language generation.