Score
Designs and conducts quantitative and qualitative analyses of how AI systems affect human decision-making and collaboration, including experiments, observational studies, and metrics to measure behavioral changes, decision-process disruptions, and collaboration dynamics. Builds measurement instruments and analytic pipelines to quantify AI influence on choices, interaction patterns, and downstream impacts such as fairness or workflow outcomes.
Current human-AI collaboration (HAIC) evaluation lacks a unified framework capable of accommodating the heterogeneity and dynamic reciprocity inherent across AI-centered, human-centered, and symbiotic HAIC paradigms. Method: This paper proposes the first structured evaluation framework tailored to all three HAIC modes, introducing a novel multi-dimensional evaluation decision tree that integrates quantitative and qualitative metrics—combining objective and subjective dimensions—and formally modeling dynamic reciprocity throughout the collaborative process. The framework is validated empirically across four domains—manufacturing, healthcare, finance, and education—via systematic literature review and cross-domain adaptation. Contribution/Results: Results demonstrate significant improvements in evaluation specificity, interpretability, and practical guidance value. The framework establishes a methodological foundation for scientifically measuring HAIC effectiveness, enabling rigorous, context-sensitive assessment of collaborative outcomes.
This study examines the tension between efficiency gains and researcher autonomy arising from AI-assisted analysis in qualitative research. Through in-depth interviews with 16 qualitative researchers, it comparatively analyzes acceptance and underlying mechanisms across three coding paradigms: fully manual, human-initiated AI-assisted, and AI-initiated. The study innovatively conceptualizes AI explicitly as a “supporter”—neither collaborator nor supervisor—and identifies three core determinants of adoption: efficiency enhancement, attribution of interpretive ownership, and algorithmic trust. Findings indicate broad acceptance of AI for accelerating coding and thematic analysis, yet strong consensus on human primacy in meaning-making and interpretive authority. Enhancing procedural transparency, researcher control, and structured human–AI collaboration significantly strengthens trust and mitigates bias risks. The work provides theoretical grounding and actionable guidelines for developing human-centered, accountable AI-augmented qualitative research workflows.
This study investigates how workers’ behavior, output quantity, and quality change when they are informed that their work will be evaluated by an AI rather than a human. Through an online experiment integrating large language model–based scoring, human assessment, and statistical analysis, the research provides the first empirical evidence of the subtle yet significant behavioral effects of algorithmic evaluation. The findings reveal that AI evaluation leads to a notable increase in output quantity, accompanied by a decline in per-unit quality. Although workers more frequently employ external tools—such as large language models—under AI evaluation, this behavior does not account for the observed shifts in productivity or quality. These results offer critical insights into the differential impacts of human versus algorithmic assessment on labor behavior.
This paper addresses ethical and methodological risks arising from excessive automation in AI-driven data science. Methodologically, it proposes a human–AI collaborative optimization framework grounded in a three-dimensional “Truth, Beauty, Justice” (TBJ) evaluation system, which systematically delineates application boundaries for generative, analytical, and agentic AI across the full data science lifecycle—from data processing and modeling to result interpretation. It positions AI as an augmentative tool rather than a replacement for human expertise, institutionalizing role-allocation mechanisms between humans and AI. The key contributions are: (1) the first integration of the TBJ philosophical framework into AI research governance; (2) the establishment of a VUCA-adaptive, data-scientist-centered collaboration paradigm; and (3) actionable guidelines for human–AI role assignment that jointly ensure scientific rigor, aesthetic coherence, and social justice—offering a theoretically grounded yet practically implementable ethics–technology integration framework for AI-augmented research.
This study addresses the profound transformations in user roles, workflows, and collaboration patterns within enterprise software platforms driven by artificial intelligence, which existing role frameworks—such as the BTP user type matrix—struggle to accommodate. Through 20 expert interviews and a participatory design workshop involving 24 participants, the research employs qualitative methods to investigate structural shifts in developer roles on the SAP Business Technology Platform. Findings reveal three key trends: automation of operational tasks, expanded human-AI collaboration, and increased reliance on agent-based systems. In response, the study argues for a necessary reconfiguration of role taxonomies and governance mechanisms, offering both theoretical grounding and practical guidance for designing and governing AI-native enterprise software.
As AI systems gain greater autonomy, domain experts increasingly assume supervisory roles—yet existing research lacks deep insight into their supervision experiences and the design of motivating interfaces. This study investigates expert supervision of AI scoring systems through four interdisciplinary co-design workshops, integrating thematic analysis, explainable AI (XAI) design, and collaborative prototyping. Grounded in the SMART work design theory, we propose the first empirically grounded, generalizable design framework that systematically links interface features to underlying psychological mechanisms—such as perceived meaningfulness and agency—comprising twelve principles. We identify four core requirements: clear role delineation, transparent decision rationale, tangible contribution impact, and efficient human-AI collaboration. The framework advances human-AI interaction theory and provides actionable, evidence-based guidance for designing interfaces that support effective and intrinsically motivating human oversight across domains.
This study addresses the risk that overreliance on artificial intelligence automation in qualitative research may undermine the depth of meaning-making. Drawing on interdependence theory, the authors propose a novel “productive interdependence” paradigm and develop a framework integrating levels of automation (LoA), task risk, and validation cost to guide human–AI collaboration across analytical stages. The framework establishes three design principles to ensure the central role of human researchers in interpretive sensemaking. Case studies demonstrate that this approach effectively calibrates trust between humans and AI, enhancing analytical efficiency while preserving the rigor and interpretive depth essential to qualitative inquiry.
This study investigates the impact of generative artificial intelligence (GenAI) tools on team creativity and collaboration in software development—a domain inherently collaborative yet increasingly mediated by individually oriented AI technologies. Through semi-structured interviews with 13 software engineers across four companies, the research employs qualitative thematic analysis to uncover an emergent triadic collaboration paradigm among developers, colleagues, and AI. Findings reveal that while GenAI expands avenues for idea generation, it may concurrently suppress independent ideation; developers often prioritize consulting AI over seeking input from teammates, thereby diminishing peer learning opportunities. Crucially, affective dynamics serve as a pivotal cross-cutting dimension, permeating both creative and collaborative processes. These insights offer theoretical and practical implications for understanding how AI reshapes collaborative mechanisms within real-world software engineering teams.
As AI agents become deeply integrated into core enterprise workflows, designing effective human-AI interaction to enhance user experience, foster adoption, and support user-centered decision-making has emerged as a critical challenge. This study addresses this issue through a mixed-methods approach, combining qualitative interviews and quantitative experiments to systematically investigate interaction patterns between humans and AI agents in business contexts and identify key design elements that shape user experience. Grounded in empirical findings, the work proposes a set of human-AI interaction design guidelines tailored for commercial environments, along with a quantifiable evaluation framework. These contributions offer both theoretical grounding and practical guidance for development teams seeking to optimize and deploy large-scale human-AI collaborative systems.