CREW: Facilitating Human-AI Teaming Research

2024-07-31

arXiv.org

AI Summary

Existing research platforms predominantly focus on simplified tasks or single-perspective analysis, limiting their capacity to support interdisciplinary empirical studies of complex human-AI collaborative decision-making. To address this gap, we introduce CREW—an open-source platform featuring a novel modular architecture designed for ecologically valid collaborative scenarios. CREW integrates cognitive experimental paradigms, real-time multimodal physiological signal acquisition (EEG, ECG, EMG), human-guided reinforcement learning benchmarks (PPO, SAC), and an extensible task framework—unifying human behavioral modeling and AI algorithm evaluation. CREW is the first tool enabling real-time, multidisciplinary, high-ecological-validity studies of human-AI teams. Within one week, we validated the platform across 50 participants: results demonstrate significant improvements in task flexibility, depth of human engagement, and rigor of algorithmic assessment. CREW thus establishes a unified experimental foundation for both foundational research and applied development in human-AI collaborative decision-making.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce CREW, a platform to facilitate Human-AI teaming research in real-time decision-making scenarios and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.

Problem

• Human-AI Collaboration
• Complex Decision Tasks
• Limitations of Current Platforms

Innovation

• CREW platform
• Human-AI collaboration
• Physiological signal recording