Assessing Code Understanding in LLMs

2025-03-31

AI Summary

Large language models (LLMs) exhibit limited understanding of compiler-level semantics-preserving program transformations—e.g., copy propagation and constant folding—critical for reliable code reasoning. Method: We propose a formal-verification–based empirical evaluation framework for semantic equivalence judgment, leveraging LLVM and other compiler toolchains to automatically generate robust test cases and self-supervised training signals. Contribution/Results: Experiments reveal high failure rates: 41% without context and 29% even with simple generic context—exposing fundamental blind spots in deep code semantic modeling. To address this, we introduce the first LLM–compiler co-enhanced training paradigm, wherein compiler-generated semantic equivalence pairs explicitly reinforce model robustness. This work establishes a rigorous methodology for quantitatively assessing code understanding capabilities and provides a scalable, tool-integrated pathway toward trustworthy code AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present an empirical evaluation of Large Language Models in code understanding associated with non-trivial, semantic-preserving program transformations such as copy propagation or constant folding. Our findings show that LLMs fail to judge semantic equivalence in approximately 41% of cases when no context is provided and in 29% when given a simple generic context. To improve accuracy, we advocate integrating LLMs with code-optimization tools to enhance training and facilitate more robust program understanding.

Problem

• Evaluating LLMs' ability to understand transformed code semantics
• Identifying LLMs' failure rates in semantic equivalence judgments
• Proposing integration with code-optimization tools for improved accuracy

Innovation

• Empirical evaluation of LLMs in code understanding
• Integration with code-optimization tools for accuracy
• Enhanced training for robust program understanding