Quo Vadis, HCOMP? A Review of 12 Years of Research at the Frontier of Human Computation and Crowdsourcing

2025-04-02

AI Summary

The rise of generative AI challenges the “human-centered” foundational premise of human computation and crowdsourcing (HC), raising theoretical questions about whether a Kuhnian paradigm shift is occurring. Method: This paper conducts the first systematic bibliometric analysis and topic modeling of 12 years (2012–2023) of HCOMP conference papers, grounded in Kuhn’s paradigm theory, augmented by cross-conference comparisons and quantitative measurement of interdisciplinary overlap. Contribution/Results: Findings indicate HC research evolves continuously rather than undergoing a discontinuous paradigm shift; thematic focus shifts significantly toward human-AI collaboration and AI-augmented crowdsourcing; and interdisciplinary coupling with AI, HCID, and related fields strengthens progressively. This study establishes the first data-driven paradigmatic benchmark for theorizing HC’s conceptual positioning and future trajectory.

Technology Category

• Humans and AI: Crowd Sourcing and Human Computation
• Application Domains: Humanities & Computational Social Science
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Web Mining and Content Analysis: Models for Web evolution
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

The field of human computation and crowdsourcing has historically studied how tasks can be outsourced to humans. However, many tasks previously distributed to human crowds can today be completed by generative AI with human-level abilities, and concerns about crowdworkers increasingly using language models to complete tasks are surfacing. These developments undermine core premises of the field. In this paper, we examine the evolution of the Conference on Human Computation and Crowdsourcing (HCOMP) - a representative example of the field as one of its key venues - through the lens of Kuhn's paradigm shifts. We review 12 years of research at HCOMP, mapping the evolution of HCOMP's research topics and identifying significant shifts over time. Reflecting on the findings through the lens of Kuhn's paradigm shifts, we suggest that these shifts do not constitute a paradigm shift. Ultimately, our analysis of gradual topic shifts over time, combined with data on the evident overlap with related venues, contributes a data-driven perspective to the broader discussion about the future of HCOMP and the field as a whole.

Problem

• Analyzing HCOMP's 12-year research evolution in human computation
• Assessing impact of generative AI on crowdsourcing field premises
• Evaluating topic shifts without paradigm change in HCOMP

Innovation

• Reviewing 12 years of HCOMP research
• Analyzing topic shifts using Kuhn's paradigm
• Assessing AI impact on human computation