LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

2026-08-06

AI Summary

This work addresses the pervasive over-reliance on Python by large language models (LLMs) in project-level code generation and the absence of systematic evaluation of their programming language selection behavior. We introduce LangChoiceBench, a benchmark spanning 28 projects across seven software domains, which enables the first systematic quantification of LLMs’ language preferences and diversity. Through manual analysis of 9,826 inference trajectories, we uncover novel failure modes, including “hallucinated evidence” and inconsistencies between recommended and implemented languages. Evaluation of 25 prominent LLMs reveals that most models select languages based on habitual bias rather than project requirements, with smaller models exhibiting even poorer language diversity, thereby highlighting a critical limitation of current LLMs in cross-language project generation.

Abstract

Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behaviour across new models. To bridge this gap, we introduce LangChoiceBench, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity. LangChoiceBench covers 28 projects across seven software areas where Python is often a poor default. We evaluate 25 diverse LLMs and find that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity. We further analyse 9,826 reasoning traces and find that most Python choices are automatic or driven primarily by ease, rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python - a failure mode we call phantom evidence - or produce code that contradicts the language selected in their own reasoning.

Problem

• programming-language choice
• Python preference
• language diversity
• recommendation-implementation consistency
• phantom evidence

Innovation

• LangChoiceBench
• programming-language choice
• recommendation-implementation consistency
• language diversity
• phantom evidence