Mind the Gap: The Missing Features of the Tools to Support User Studies in Software Engineering

2025-04-11

AI Summary

Software engineering user studies face multiple persistent barriers, yet existing tool support remains critically inadequate. This study systematically identifies tool coverage gaps through barrier–function mapping analysis, functional auditing of 32 tools, a large-scale expert survey (N=102), and rigorous statistical testing. Results reveal that critical barriers—including cross-cultural participant recruitment and dynamic ethics approval—receive zero tool support; this work provides the first quantitative confirmation of complete coverage absence for several core barriers. Based on these findings, we propose ten novel functionality requirements, nine of which achieve statistical significance (p < 0.01) in expert validation. Our contribution is twofold: (1) uncovering fundamental structural deficits in the current tool ecosystem, and (2) establishing an empirically grounded, function-centric design framework to guide the development of next-generation user research support tools for software engineering.

Technology Category

• Humans and AI: User Experience and Usability
• Application Domains: Humanities & Computational Social Science
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

User studies are paramount for advancing science. However, researchers face several barriers when performing them despite the existence of supporting tools. In this work, we study how existing tools and their features cope with previously identified barriers. Moreover, we propose new features for the barriers that lack support. We validated our proposal with 102 researchers, achieving statistically significant positive support for all but one feature. We study the current gap between tools and barriers, using features as the bridge. We show there is a significant lack of support for several barriers, as some have no single tool to support them.

Problem

• Identify gaps in tools for software engineering user studies
• Propose new features to address unsupported research barriers
• Validate feature proposals with researcher feedback

Innovation

• Analyzing tool-barrier gaps via features
• Proposing new features for unsupported barriers
• Validating features with researcher feedback