LLM-Based Program Generation for Triggering Numerical Inconsistencies Across Compilers

2025-08-29

AI Summary

Floating-point computations exhibit inconsistent results across compilers—including cross-platform and cross-optimization-level settings—posing a severe threat to the reliability of numerical software. This paper introduces, for the first time, large language models (LLMs) into floating-point inconsistency detection, proposing a synergistic approach that integrates syntax-guided program generation with feedback-driven mutation to automatically synthesize floating-point programs with high inconsistency-triggering capability. The method enables systematic comparative testing across multiple compilers, optimization levels, and heterogeneous host-device environments. Experimental evaluation demonstrates that our approach detects over twice as many floating-point inconsistencies as state-of-the-art tools. Moreover, it is the first to systematically uncover deep real-valued discrepancies across architectures and compiler toolchains, significantly enhancing verifiability and robustness assurance for numerical software.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Floating-point inconsistencies across compilers can undermine the reliability of numerical software. We present LLM4FP, the first framework that uses Large Language Models (LLMs) to generate floating-point programs specifically designed to trigger such inconsistencies. LLM4FP combines Grammar-Based Generation and Feedback-Based Mutation to produce diverse and valid programs. We evaluate LLM4FP across multiple compilers and optimization levels, measuring inconsistency rate, time cost, and program diversity. LLM4FP detects over twice as many inconsistencies compared to the state-of-the-art tool, Varity. Notably, most of the inconsistencies involve real-valued differences, rather than extreme values like NaN or infinities. LLM4FP also uncovers inconsistencies across a wider range of optimization levels, and finds the most mismatches between host and device compilers. These results show that LLM-guided program generation improves the detection of numerical inconsistencies.

Problem

• Detects floating-point inconsistencies across different compilers
• Generates programs to trigger numerical software reliability issues
• Identifies real-valued differences in compiler optimization levels

Innovation

• LLM-based program generation for compiler inconsistencies
• Grammar-based generation and feedback-based mutation
• Detects twice as many inconsistencies as state-of-the-art