formal problem definition

This work addresses critical shortcomings in current AI benchmark tasks—such as insufficient correctness, ambiguous solvability, verification difficulties, unclear objectives, and unrealistic difficulty levels—by formally introducing, for the first time, five core attributes of a “good task.” Integrating task design theory with software engineering practices, the authors develop a qualitative analytical framework that emphasizes grounding tasks in real-world scenarios, articulating them in practitioners’ language, and validating outcomes based on results rather than methods. The resulting normative guidelines embed practitioner perspectives directly into benchmark design, substantially enhancing the validity and real-world relevance of evaluation tasks. This contribution provides both theoretical foundations and practical standards for developing high-quality AI benchmarks.

VDM tools have long lacked automated generation of proof obligations for explicit operation bodies. Method: This paper introduces the first proof obligation generation framework targeting VDM operational semantics, built upon the formal semantics of VDM-SL, first-order logic encoding, and interfaces to theorem provers. It enables end-to-end derivation from operation definitions to verifiable logical assertions. The framework features a novel structured operation expansion mechanism and a synthesis algorithm for pre- and postconditions, supporting core constructs—including sequential composition, conditionals, loops, and function calls—with 85% coverage. Contribution/Results: This work fills a critical gap in the VDM toolchain for operation-level automated verification, significantly enhancing both the efficiency and trustworthiness of model verification.

Automating the formalization of research-level mathematical theorems in the Lean proof assistant remains challenging due to the gap between abstract mathematical structures and their concrete instantiations. Method: We propose a structured, template-driven approach that bridges this gap systematically. It employs reusable, modular templates to explicitly encode mappings from abstract structures to concrete instances; leverages large language models to generate candidate definitions and theorems; utilizes Lean’s type-class mechanism for automatic instance resolution; and incorporates structural hypothesis verification and feedback-guided iterative refinement to ensure formal correctness. Contribution/Results: This work achieves the first end-to-end automated formalization of theorems across multiple concrete instances derived from a single abstract structure. Evaluated on an optimization-theory dataset, our method successfully generated multiple correct, machine-verifiable Lean proofs. It significantly improves both the efficiency and breadth of mathematical formalization, advancing scalable, reliable automation in interactive theorem proving.

This work addresses the lack of formal guarantees regarding semantic preservation during problem reformulation and solver correctness in constraint programming. It presents the first end-to-end verified framework implemented in the Lean theorem prover, enabling formal proofs of parameterized equivalence, equisatisfiability, and symmetry-breaking correctness for entire families of problems. The approach combines general, parameterized proofs with instance-level certificate checking, thereby eliminating the need to trust external solvers. Verified certificates are produced via backend transformations, and a single high-level proof suffices for arbitrarily large instances. This methodology achieves dramatic search-space reductions—up to a factor of twenty million—and enables full verification of the largest instances in just a few minutes.

Software engineering research has long lacked a structured methodology to guide researchers in formulating industrially relevant research questions. Method: This paper proposes and empirically validates a seven-dimensional problem modeling framework—comprising Actual Problem, Context, Impact, Practitioners, Evidence, Goal, and Research Question—and innovatively incorporates financial dimensions (e.g., ROI) and feasibility constraints to enhance industrial applicability. We conducted an empirical evaluation with 42 senior SE researchers via participatory workshops using Problem Vision boards, structured questionnaires, and qualitative analysis. Contribution/Results: The framework significantly improves the practical relevance and operationalizability of research questions. It yields actionable guidelines for refining problem formulation, thereby effectively bridging the gap between academic research and industrial needs.

Existing statistical model checking methods suffer from insufficient theoretical foundations and limited verification reliability. This work establishes the first comprehensive probabilistic-logical formal framework for the SCAN statistical model checker, integrating probabilistic model checking, statistical hypothesis testing, and formal verification techniques to rigorously characterize the property verification process of complex systems. By unifying these complementary approaches within a sound theoretical basis, the proposed framework not only addresses the foundational gaps previously present in SCAN but also significantly enhances its rigor and applicability. Consequently, it provides a robust guarantee for the reliability of SCAN when applied to the verification of real-world systems.

This work addresses the challenge of formal verification for Reflex programs in industrial-scale control systems, where the generation of an excessive number of verification conditions often renders manual analysis impractical. To overcome this limitation, the authors propose a hybrid verification strategy that integrates a structured requirement annotation language with automated invariant inference based on program structure, coupled with an SMT solver to automatically discharge a substantial subset of verification conditions. By leveraging this synergistic approach, the method significantly reduces the number of verification tasks requiring human intervention, thereby enhancing the automation, feasibility, and overall efficiency of formal verification for large-scale process control systems.

This work addresses the challenge that counterexamples generated by formal verification often consist of numerous low-level Boolean variables, rendering them difficult for developers to interpret at the application-domain level. To bridge this gap, the paper proposes a novel hierarchical explanation method that integrates predicate relevance metrics with dependency graph analysis—a first-time fusion of these two techniques—to automatically extract human-readable, domain-oriented explanations from logical formulas. By leveraging formal modeling and a dedicated explanation-generation algorithm, the approach produces concise and semantically clear descriptions of failure causes across multiple case studies. Empirical results demonstrate that the method significantly outperforms existing techniques, offering effective support for fault localization in practical verification tasks.

This work addresses the challenge of irreproducibility in data analysis scripts, which often stems from implicit assumptions—such as specific package versions, expected data formats, or undocumented manual interventions. The paper proposes a static analysis approach tailored to data analysis workflows that, for the first time, unifies diverse implicit assumptions into inferable constraint models. By leveraging customized program analysis and example-driven modeling, the authors develop a prototype system capable of automatically identifying these hidden assumptions, extracting executable preconditions, and generating verifiable constraints. The resulting framework supports runtime validation and automatic documentation generation, substantially enhancing script executability, reproducibility, and interpretability.

This study addresses the prevalent ambiguity, inconsistency, and incompleteness in articulating explainability requirements for AI systems due to a lack of standardized specifications. Through a structured literature review and interviews with developers, the authors identify a set of explainability quality attributes, which are then refined via a large-scale survey of practitioners into ten core attributes. For the first time, these attributes are translated into a prioritized, actionable guideline for writing explainability requirements. Building on this foundation, the authors design a lightweight, iterative requirements engineering workflow augmented by a large language model to assist in requirement generation. An accompanying web-based tool reduces average requirement drafting time by 23.5%, and user evaluations indicate that the generated requirements match or slightly exceed manually written ones in terms of implementability and textual quality.