An Agent-Based Framework for the Automatic Validation of Mathematical Optimization Models

2025-11-20

AI Summary

Mathematical optimization models generated by large language models (LLMs) often suffer from unverifiable semantic correctness and constraint consistency. Method: This paper proposes the first multi-agent automated verification framework specifically designed for optimization modeling. It innovatively adapts mutation testing—a software engineering technique—to the optimization domain by introducing a domain-specific testing API and orchestrating multiple LLM agents to collaboratively generate test cases, construct model mutants, and assess semantic consistency. The approach integrates LLMs, multi-agent coordination, automated testing API generation, and optimization-aware mutation strategies, while leveraging test coverage metrics to quantify verification efficacy. Contribution/Results: Experiments demonstrate that the framework achieves significantly higher mutation coverage than baseline methods, effectively detects modeling errors, and substantially enhances the reliability and trustworthiness of LLM-generated optimization models.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recently, using Large Language Models (LLMs) to generate optimization models from natural language descriptions has became increasingly popular. However, a major open question is how to validate that the generated models are correct and satisfy the requirements defined in the natural language description. In this work, we propose a novel agent-based method for automatic validation of optimization models that builds upon and extends methods from software testing to address optimization modeling . This method consists of several agents that initially generate a problem-level testing API, then generate tests utilizing this API, and, lastly, generate mutations specific to the optimization model (a well-known software testing technique assessing the fault detection power of the test suite). In this work, we detail this validation framework and show, through experiments, the high quality of validation provided by this agent ensemble in terms of the well-known software testing measure called mutation coverage.

Problem

• Validating LLM-generated optimization models against natural language requirements
• Automating correctness verification for mathematical optimization model generation
• Extending software testing techniques to assess optimization model quality

Innovation

• Agent-based framework for automatic optimization model validation
• Generates problem-level testing API and model-specific mutations
• Extends software testing techniques to mathematical optimization models