CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments

2025-10-31

AI Summary

Existing code generation benchmarks overemphasize functional correctness while neglecting developers’ practical needs regarding code style, constraint adherence, and iterative refinement. Method: We introduce the first developer-preference-oriented, multilingual instruction-following benchmark, featuring a dual-scenario evaluation framework—“initial constraint satisfaction” and “subsequent instruction response”—covering Python, Java, and JavaScript. Built upon LiveBench tasks, it leverages high-quality automated translation and an end-to-end automated evaluation pipeline to assess models across syntax compliance, semantic consistency, constraint fidelity, and iterative modification capability. Contribution/Results: Our systematic evaluation reveals substantial cross-lingual and cross-task disparities in instruction-following performance among state-of-the-art large language models. The benchmark delivers a fine-grained, development-context-aware performance profile—significantly advancing beyond traditional correctness-centric metrics and better reflecting real-world coding workflows.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

As large language models become increasingly capable of generating code, evaluating their performance remains a complex and evolving challenge. Existing benchmarks primarily focus on functional correctness, overlooking the diversity of real-world coding tasks and developer expectations. To this end, we introduce a multi-language benchmark that evaluates LLM instruction-following capabilities and is extensible to operate on any set of standalone coding problems. Our benchmark evaluates instruction following in two key settings: adherence to pre-defined constraints specified with the initial problem, and the ability to perform refinements based on follow-up instructions. For this paper's analysis, we empirically evaluated our benchmarking pipeline with programming tasks from LiveBench, that are also automatically translated from Python into Java and JavaScript. Our automated benchmark reveals that models exhibit differing levels of performance across multiple dimensions of instruction-following. Our benchmarking pipeline provides a more comprehensive evaluation of code generation models, highlighting their strengths and limitations across languages and generation goals.

Problem

• Assessing code generation models on developer-preferred code adjustments
• Evaluating LLM instruction-following capabilities across multiple languages
• Measuring adherence to constraints and refinement based on follow-up instructions

Innovation

• Multi-language benchmark for code generation models
• Evaluates instruction-following with constraints and refinements
• Automated pipeline tests across languages and generation goals