operationalization design

Data scientists frequently lack systematic guidance when operationalizing ambiguous concepts (e.g., “writing authenticity,” “medical need”) into model-ready proxy target variables. To address this, we conducted semi-structured interviews with 15 data scientists across education and healthcare domains, followed by cross-domain thematic coding. We propose the “assemblage metrics” framework, identifying five core design criteria: validity, simplicity, predictiveness, portability, and resource efficiency. Our analysis reveals an iterative, problem-reconstruction–driven practice in which target variables are dynamically negotiated through trade-offs among these criteria. This work offers the first systematic characterization of such trade-offs in proxy target construction. It contributes a theoretically grounded framework and methodological tools for HCI, CSCW, and machine learning communities to support principled, transparent, and trustworthy predictive modeling—bridging conceptual abstraction with operationalizable measurement.

Current approaches to automated program synthesis lack effective governance mechanisms to ensure the compliance of generated code. This work proposes Protocol-Driven Development (PDD), a model that treats machine-executable protocols as primary artifacts and delineates the space of valid implementations through structural, behavioral, and operational invariants. PDD mandates that every implementation be accompanied by a verifiable chain of compliance evidence. By integrating formal methods, property-based testing, policy-as-code, and software provenance techniques, PDD establishes a unified framework for protocol specification and verification. This framework enables trustworthy admission control over automatically synthesized code, guaranteeing that all adopted implementations strictly adhere to protocol constraints and are backed by complete, auditable proofs of compliance.

This work addresses the prevailing lack of systematic understanding of foundational formal theories in current AI compiler design, which hinders rigorous evaluation of the completeness and desirability of intermediate representations and compilation abstractions. For the first time, it systematically establishes precise correspondences between core mechanisms of MLIR—such as term rewriting systems, refinement calculi, and abstract interpretation—and classical formal theories. By grounding compiler abstractions in formal semantics, the paper clarifies the theoretical underpinnings of these constructs, articulates a precise notion of “design completeness,” and provides assessable criteria and guiding principles to navigate trade-offs between engineering pragmatism and theoretical ideals.

This work addresses the lack of a unified semantic foundation in current software systems, which creates comprehension gaps among development, usage, and governance due to deficiencies in usability, modularity, and accountability. To bridge this divide, the paper proposes grounding software semantics in domain behavioral phenomena—specifically individuals, actions, and facts—as a shared conceptual vocabulary for stakeholders. This approach systematically integrates phenomenon-based modeling into software development by organizing behaviors into conceptual units, leveraging large language models (LLMs) to map semantics to modular, readable code, and establishing agent accountability through behavior-oriented norms. Empirical evaluation demonstrates that the proposed method significantly enhances the quality of usability design, improves the modularity and readability of LLM-generated code, and strengthens the accountability of autonomous agent behaviors.

Scientists frequently record experimental metadata in spreadsheets, yet ensuring consistency and standards compliance remains challenging. This paper introduces a spreadsheet-native metadata governance paradigm: customized Excel/CSV templates embed HuBMAP standards; OWL/SKOS ontology-driven controlled vocabularies are integrated; and a web-based real-time semantic validation tool enables immediate, on-entry verification. The approach seamlessly incorporates semantic constraints into familiar spreadsheet workflows—requiring no platform switching or new system adoption. Deployed across the HuBMAP Consortium, it significantly improved multi-omics metadata compliance rates, increased data entry efficiency, and reduced error identification and correction time by over 70%. To our knowledge, this is the first work to deeply embed ontology-based constraints and real-time semantic validation directly within spreadsheet environments, establishing a scalable, practical paradigm for biomedical metadata standardization.

This study addresses a critical limitation in traditional reproducible research, where sharing only code and results fails to expose the implicit assumptions, expectations, and premises underlying an analyst’s reasoning—thereby hindering thorough evaluation of analytical quality. To overcome this, the paper proposes a formal modeling framework that explicitly translates the analyst’s tacit reasoning process into structured logical representations, statically capturing the construction logic of the analysis. This approach enables systematic scrutiny of the analytical chain of reasoning, assumption sensitivity, and conclusion robustness—even in the absence of the original data. Empirical validation on representative data analysis tasks demonstrates the framework’s effectiveness, achieving both logical visualization and data-free static assessment of analytical integrity.

This study addresses the persistent challenge in software engineering research of empirically validating theories due to the absence of systematic, reproducible operationalization methods. To bridge this gap, the authors propose an integrated methodological framework that combines Sjøberg’s operationalization approach with Dubin’s theory-building methodology, offering the first evidence-driven and replicable guide for operationalizing theoretical constructs in software engineering. The approach systematically translates abstract theories into measurable forms by rigorously defining variables, selecting appropriate indicators, and deriving non-causal assumptions. The utility of the framework is demonstrated through its application to a theory on DevOps team classification. The resulting methodology provides researchers with a robust foundation for conducting verifiable theoretical studies while simultaneously offering practitioners actionable, theory-informed insights.

Existing approaches to automatic formalization are largely confined to isolated statements and struggle to capture the intricate dependency structures among axioms, definitions, and lemmas within mathematical theories. This work introduces a novel paradigm—*theory-level automatic formalization*—which systematically advocates shifting from statement-level to integrated theory-level formalization. By constructing formalized mathematical libraries, modeling dependency graphs, and establishing mappings from natural to formal languages, the approach enables machine-verifiable translation of entire mathematical theories along with their internal structures. The paper delineates core challenges inherent to this direction, proposes three viable research pathways, and releases a comprehensive survey resource to catalyze further progress in the field.

Although large language models (LLMs) can achieve agreement with human annotators in text coding, their judgments may rely on superficial features unrelated to the underlying theoretical construct, thereby lacking construct validity. To address this issue, this work proposes a “fine-grained calibration” approach that decomposes theoretical constructs into clause-level components, validates each component against extractive evidence, and aggregates results according to explicit theoretical rules to assess whether LLMs genuinely measure the target construct. This method shifts the validation of construct validity from output consistency to process interpretability, enabling identification of errors stemming either from missing components or confusion with neighboring constructs. It establishes a transparent and interpretable paradigm for trustworthy measurement using LLMs in the social sciences.