operational definition

Current AI system evaluations suffer from fragmented assessment dimensions, heterogeneous evidence sources, and insufficient transparency. To address these challenges, this paper proposes the “Measurement Tree”—a novel multi-source fusion evaluation framework based on a hierarchical directed graph. Structured as a tree-like data model, it supports user-defined aggregation functions to unify heterogeneous metrics—including agency, business value, energy efficiency, socio-technical impact, and safety—into interpretable, multi-level representations. This work introduces, for the first time, a hierarchical graph structure as the formal output format for AI evaluation, substantially enhancing traceability and interpretability. An accompanying open-source Python library and extensive empirical validation demonstrate that the Measurement Tree improves comprehensiveness, operationality, and reproducibility in evaluating complex AI systems. It thus provides foundational infrastructure for building an open and transparent AI evaluation ecosystem.

This work addresses the lack of a unified semantic foundation in current software systems, which creates comprehension gaps among development, usage, and governance due to deficiencies in usability, modularity, and accountability. To bridge this divide, the paper proposes grounding software semantics in domain behavioral phenomena—specifically individuals, actions, and facts—as a shared conceptual vocabulary for stakeholders. This approach systematically integrates phenomenon-based modeling into software development by organizing behaviors into conceptual units, leveraging large language models (LLMs) to map semantics to modular, readable code, and establishing agent accountability through behavior-oriented norms. Empirical evaluation demonstrates that the proposed method significantly enhances the quality of usability design, improves the modularity and readability of LLM-generated code, and strengthens the accountability of autonomous agent behaviors.

Existing SQL lacks native support for efficient, composable, and semantically closed multidimensional analysis in business intelligence (BI), hindering query reusability, maintainability, and expressive power. Method: This paper introduces a SQL-native “measure column” mechanism that embeds context-sensitive expressions and dynamically scoped evaluation contexts directly into relational table structures. Leveraging syntactic extensions and compile-time expansion, it enables composability, expandability, and strict adherence to standard SQL semantics—every measure invocation compiles losslessly into equivalent standard SQL without runtime interpretation. Contribution/Results: This is the first approach to support semantically closed multidimensional computation natively within standard SQL. It bridges the flexibility of multidimensional languages with the precision and engineering robustness of SQL, significantly enhancing BI query reuse, maintainability, and expressiveness while preserving full compatibility with existing SQL tooling and execution engines.

This paper addresses the challenge of characterizing behavioral distances in quantitative systems—such as probabilistic and fuzzy transition systems—using modal logic, where existing logical frameworks lack precise expressive power. Methodologically, we first prove that probabilistic trace distance is not definable by unary modal logic; then introduce a graded monadic algebraic construction to establish general criteria for real-valued modal logics to characterize behavioral distances; finally, leveraging category theory and coalgebraic techniques, we develop a novel characteristic logic for trace distance in fuzzy metric transition systems. Our main contribution is a unified, scalable, and multi-granular framework for behavioral distances that overcomes the expressive limitations of classical modal logic. This framework constitutes the first metric semantic unification for quantitative transition systems—spanning both probabilistic and fuzzy settings—with rigorous theoretical foundations and broad modeling applicability.

Current approaches to automated program synthesis lack effective governance mechanisms to ensure the compliance of generated code. This work proposes Protocol-Driven Development (PDD), a model that treats machine-executable protocols as primary artifacts and delineates the space of valid implementations through structural, behavioral, and operational invariants. PDD mandates that every implementation be accompanied by a verifiable chain of compliance evidence. By integrating formal methods, property-based testing, policy-as-code, and software provenance techniques, PDD establishes a unified framework for protocol specification and verification. This framework enables trustworthy admission control over automatically synthesized code, guaranteeing that all adopted implementations strictly adhere to protocol constraints and are backed by complete, auditable proofs of compliance.

This work addresses the prevailing lack of systematic understanding of foundational formal theories in current AI compiler design, which hinders rigorous evaluation of the completeness and desirability of intermediate representations and compilation abstractions. For the first time, it systematically establishes precise correspondences between core mechanisms of MLIR—such as term rewriting systems, refinement calculi, and abstract interpretation—and classical formal theories. By grounding compiler abstractions in formal semantics, the paper clarifies the theoretical underpinnings of these constructs, articulates a precise notion of “design completeness,” and provides assessable criteria and guiding principles to navigate trade-offs between engineering pragmatism and theoretical ideals.

This work addresses the risk that online platforms may strategically generate semantically equivalent content variants to manipulate compliance metrics, creating a “gaming” problem where apparent metric improvements mask unmitigated harms. The authors model moderation protocols as transformation graphs and introduce a semantic envelope metric, theoretically proving it to be the pointwise minimal solution within the class of conservative repairs. They further develop a hierarchical certification mechanism that guarantees effective constraint of true harm under any policy. Experimental evaluation—combining finite-state mixed-strategy enumeration, SMT solving (using Z3 and cvc5), and bounded single-player MDP verification in PRISM-games—demonstrates that conventional metrics often exhibit significant violations and gaming gaps, whereas the semantic envelope metric remains violation-free across all test instances, effectively resisting strategic manipulation.

This work proposes a novel reflective protocol that enables runtime upward navigation through the semantic tower—a capability absent in existing runtime systems, which are restricted to downward execution along abstraction layers. By formally integrating operational semantics with runtime reflection, the approach introduces the notion of “first-class implementations” and establishes a generalized safe-point mechanism grounded in formal specifications. This mechanism permits observation and dynamic switching of high-level abstractions’ underlying implementations during execution, thereby overcoming the traditional limitation of supporting only downward compilation or interpretation. The study demonstrates, for the first time, the feasibility and effectiveness of deeply integrating semantic theory with runtime system design, enabling adaptive and semantically aware execution environments.

Traditional algebraic rewriting is unreliable for expressions involving measurements due to domain inconsistencies arising from repeated observations and division operations. This work proposes a unified semantic framework that simultaneously tracks both the provenance and definedness of expressions, enabling sound one-way rewriting and interchangeability judgments. By introducing label-sensitive bracketing semantics, admissible domain refinement, and a relative variant of support sets, the authors develop domain-safe rewriting rules and formally prove restoration and strictness theorems. All results are fully formalized in Lean 4 without any use of `sorry`, revealing fundamental limitations: simplifications are generally irreversible, equivalence over a common domain is insufficient, and label erasure inherently causes information loss.

This work addresses the challenge that domain experts face in translating natural language descriptions of data quality requirements into executable analyses, a process often hindered by reliance on data engineers, resulting in inefficiency and high technical barriers. To overcome this, the paper proposes a no-code, model-driven pipeline that leverages a QPM metamodel to define domain-specific quality analysis templates. Coupled with the Constrainify toolchain, it automatically transforms natural language requirements into executable and reusable analytical logic. By integrating model-driven engineering, metamodeling, and no-code web technologies, the approach significantly reduces dependency on technical expertise, enabling efficient, reproducible, and semantically aligned data quality assessments. This advancement enhances both the accessibility and automation of data quality analysis for non-technical domain practitioners.