Verified LLM-Driven Synthesis for Concept Design

2026-07-17

AI Summary

This work addresses the challenge in conceptual design where relying solely on safety invariants often fails to uniquely determine reaction rules aligned with user intent, leading to inconsistent synthesis outcomes. To overcome this, the authors propose a novel synthesis approach that integrates formal semantics with an LLM-driven Counterexample-Guided Inductive Synthesis (CEGIS) framework. The method leverages either positive/negative example scenarios or natural language prompts to guide the generation of rules satisfying given safety invariants, and introduces, for the first time, an LLM-assisted scenario-based elicitation mechanism to support early-stage design exploration. As the first effort to combine formal verification with LLM-based synthesis in conceptual design, experiments demonstrate that scenario-based guidance more reliably reproduces intended designs than natural language alone; with sufficient scenarios, LLM-augmented elicitation effectively recovers expected behaviors for most variants, though behavior omission and non-determinism remain key obstacles to achieving full coverage.

Abstract

Concept Design structures software systems around concepts: user-facing, self-contained units of functionality with a focused purpose. Concepts are composed into applications using synchronization rules called reactions, which specify how actions in one concept trigger actions in others. This paper first gives a formal semantics for concepts and reactions, enabling automatic verification of safety invariants in applications developed with this methodology. It then presents a CEGIS-style, LLM-driven synthesis procedure for generating reaction designs that satisfy such invariants. Because many different designs can satisfy the same invariant, we study two ways of steering synthesis toward the user's intended design: natural-language prompts and positive/negative scenarios. We also propose an LLM-driven scenario elicitation technique to support early design exploration. In an evaluation on three applications and twelve design variants using one LLM configuration, invariant-only synthesis reached verified designs quickly but often produced inconsistent designs across runs, some of which were implausible, showing that invariants alone underconstrain the design task. Scenario-guided synthesis recovered intended designs more consistently than natural-language prompting, although minimal scenarios can lead to overfitting. LLM-driven scenario elicitation, where the user classifies proposed scenarios rather than authoring them from scratch, recovered the intended designs in most variants when enough scenarios were elicited, but missed behaviors and non-determinism prevented reliable coverage in all cases.

Problem

• concept design
• reaction synthesis
• safety invariants
• design intent
• LLM-driven synthesis

Innovation

• Concept Design
• LLM-driven synthesis
• Formal verification
• Reaction synthesis
• Scenario elicitation