Enhancing Code Generation for Low-Resource Languages: No Silver Bullet

2025-01-31

AI Summary

This study addresses the degraded code generation performance of large language models (LLMs) for niche programming languages—such as R and Racket—caused by scarce training data. We systematically evaluate three adaptation strategies: fine-tuning, in-context learning (ICL), and cross-lingual pretraining. Experiments span six LLMs across diverse architectures and parameter scales. Our key finding is a strong interaction between model scale and adaptation efficacy: fine-tuning yields optimal results for small models; ICL proves more robust, cost-effective, and consistently beneficial for medium-to-large models; and fine-tuning unexpectedly harms performance for very large models. This challenges the prevailing assumptions that “larger models are universally better” and “fine-tuning is always effective,” revealing scale-dependent adaptation trade-offs. The results establish a scalable, low-cost adaptation paradigm for code generation in under-resourced programming languages.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The advent of Large Language Models (LLMs) has significantly advanced the field of automated code generation. LLMs rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages (i.e., niche programming languages characterized by the scarcity of training data), the limited availability of such data hampers the models' ability to generalize effectively, resulting in poorer code generation performance as compared to high-resource languages. For this reason, there is a quest for techniques able to close this performance gap. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages, namely: (i) a classic fine-tuning, which is however capped in size by the scarcity of training data; (ii) three variants of in-context learning, with prompts crafted to provide the LLM with additional information about the low-resource language (e.g., few-shot examples showcasing features of the targeted language); and (iii) a pre-training objective teaching the model how to translate between high- and low-resource languages. The context of our study are two low-resource languages (R and Racket) and six LLMs having different architectures and sizes. Our findings reveal that a fine-tuning is usually the best choice for smaller LLMs, possibly due to the fact that even a small dataset is sufficient to train their limited number of parameters. With the increase in size of the models, in-context learning becomes more and more effective, representing a safe and cheap bet (i.e., it always helps, but with different magnitudes). Differently, very large LLMs may deteriorate their performance on low-resource languages when fine-tuning is performed, possibly due to the lack of enough data needed to effectively update their weights.

Problem

• Large Language Models
• Niche Programming Languages
• Data Scarcity

Innovation

• Fine-tuning
• In-context Learning
• Performance Enhancement