real-world outcome evaluation

Existing performance measurement frameworks struggle to simultaneously satisfy customizability, interpretability, and mathematical tractability in interdisciplinary contexts. Method: This paper proposes a goal-oriented, customizable metric construction framework featuring a novel “base metric–auxiliary metric” dichotomy. Integrating utility theory and multi-criteria decision analysis, it introduces an uncertainty-aware utility function and establishes a systematic metric decomposition–synthesis workflow. Contributions: (1) It reduces reliance on complex mathematical formalisms, enhancing applicability under resource constraints or high uncertainty; (2) it ensures metric transparency, traceability, and domain adaptability; and (3) it enables quantitative assessment of goal attainment, real-time progress monitoring, and downstream statistical modeling and decision optimization. The framework has been empirically validated across diverse disciplines, demonstrating generality and extensibility.

Existing computational tools for qualitative data analysis often fall short in effectively supporting causal exploration due to insufficient contextual awareness, limited trustworthiness, or overly complex outputs. To address these limitations, this work proposes QualCausal, the first interactive causal analysis system grounded in user research–driven design principles. Developed through formative user studies, QualCausal integrates context-aware processing, cognitive scaffolding, and explainability mechanisms to facilitate efficient exploration and validation of causal hypotheses within qualitative datasets. The system enables researchers to extract causal relationships, construct interactive causal networks, and examine findings through coordinated multi-view visualizations. User evaluations demonstrate that QualCausal significantly reduces analytical burden, provides robust cognitive support, and prompts critical reflection on how computational tools can be meaningfully integrated into social science research practices, thereby bridging the gap between computational assistance and qualitative inquiry paradigms.

This study addresses a critical gap in the evaluation of mental health conversational AI by shifting focus beyond symptom reduction to include everyday functional outcomes and potential risks, such as inflated self-perception. Using a single-arm observational cohort design, it tracked psychological functioning over four weeks among 1,284 real-world users of the AI system “Ash.” The assessment framework was innovatively expanded to encompass daily functioning indicators—life satisfaction, interpersonal relationships, and sleep quality—while concurrently monitoring grandiosity. Longitudinal data were collected via in-app single-item scales and analyzed using paired t-tests and ANCOVA. Results revealed significant improvements across all functional domains and therapeutic alliance (p<.001, d=0.14–0.26), with no increase in grandiose tendencies. Moreover, usage intensity—measured by active days, conversation frequency, and duration—significantly predicted functional outcomes at week 4.

This study addresses the pervasive issue of measurement error in both outcome variables and multiple covariates within routinely collected biomedical data, such as electronic health records, which, if uncorrected, can induce analytical bias and misinform clinical decisions. For the first time within a tutorial framework, it systematically reviews and empirically compares several methods capable of simultaneously correcting measurement error in both outcomes and multiple covariates—including regression calibration, SIMEX, instrumental variable approaches, and modeling strategies leveraging validation subsamples. Through a unified illustrative example and publicly available code, the work not only clarifies the relative performance of these methods in real-world data to guide researchers’ methodological choices but also establishes a reproducible end-to-end analytical pipeline and highlights promising directions for future research.

Estimating the functional relationship between a continuous exposure and a binary outcome is challenging when covariates are measured with error. This study presents the first systematic evaluation of Simulation-Extrapolation, Regression Calibration, multiple imputation, and Bayesian correction methods, each coupled with flexible modeling techniques—including B-splines, P-splines, and fractional polynomials—within a multi-team, fully blinded, neutral simulation framework. By generating 155 distinct simulation scenarios and repeated samples, the research quantifies the bias and variance of each approach, revealing their relative strengths and limitations. The findings not only inform method selection under measurement error but also demonstrate the feasibility and value of this neutral comparative paradigm for rigorous methodological assessment.

This study addresses the overreliance on inter-annotator agreement in current data annotation practices, which often overlooks annotation’s capacity to capture conceptual validity as a measurement act. Treating annotation as a measurement process, the work identifies five root causes of annotation issues—errors, ambiguity, impossibility, subjectivity, and annotator identity—and develops a measurement theory–based framework for diagnosing and improving annotation quality. Drawing on a synthesis of 132 literature sources and 10 semi-structured interviews, the research systematically defines target constructs, designs annotation instruments, implements labeling procedures, and evaluates both reliability and validity. The resulting framework equips annotation teams with evaluation methods that transcend mere agreement metrics, thereby substantially strengthening the foundational quality of AI training data.

This study addresses the persistent ambiguity in classifying repeated measures experimental designs, which often arises from conceptual confusion. To resolve this issue, the authors systematically clarify the core characteristics of such designs and propose a novel classification framework grounded in experimental units and randomization strategies. For the first time in this context, Hasse diagrams are introduced to visually represent the hierarchical structure of these designs. This approach effectively distinguishes among various types of repeated measures designs, eliminates terminological ambiguities, and substantially enhances both the rigor and interpretability of experimental planning and reporting.