Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits

2026-07-18

AI Summary

This work addresses the challenge of silent errors in large language models (LLMs) when translating natural language into optimization models, which often lead to subtle semantic deviations. The authors propose a reference-free, falsification-based verification framework that generates test instances through slot perturbations and leverages solvers for error detection. Drawing on duality theory, comparative statics, and polyhedral limit arguments, they design a suite of acoustic test classes—including collapse probes, forbidden limits, annihilation, and swap tests—that jointly examine directional, curvature, and asymptotic properties. This approach achieves, for the first time, zero false positives in verifying model fidelity. Evaluated on 326 real-world models, the method attains a 0.0% false positive rate—substantially outperforming threshold-based methods (54.9%)—and detects 70.0% of conditional mutants, 40.4% of which are invisible to conventional execution-based accuracy metrics.

Abstract

Large language models now translate natural-language descriptions of decision problems into solver-ready optimization models, but they fail silently. A generated model often runs and still formulates the wrong problem. This paper develops a theory of falsification-based verification for this setting. Every numeric quantity in the description is a typed slot, and a candidate model is tested only through solver calls on slot-transformed instances; no reference model or label is consulted. From duality, comparative statics, and polyhedral limit arguments we derive a battery of test classes covering directions, curvature, crush probes, prohibitive limits, annihilation, and exchange. Every test is sound, so a violation certifies unfaithfulness and the false-positive rate is zero by design. We characterize what such verification can never see, give conditions under which the canonical error classes are detected with certainty, and prove that no fixed-threshold perturbation tester is simultaneously sound and nontrivial. Experiments on 326 ground-truth models from NL4OPT and four benchmark families confirm the theory. The battery attains a 0.0% false-positive rate against 54.9% for a threshold tester, detects 70.0% of certified conditional-class mutants, convicts 40.4% of the mutants invisible to execution-accuracy scoring, and reproduces the predicted detectability pattern including its zeros.

Problem

• falsification
• verification
• large language models
• optimization models
• sound testing

Innovation

• falsification-based verification
• optimization model generation
• sound test battery
• LLM-generated models
• detectability limits