Human-AI Interaction and User Satisfaction: Empirical Evidence from Online Reviews of AI Products

2025-03-23

AI Summary

Existing human-AI interaction (HAI) principles lack large-scale empirical validation regarding their impact on user satisfaction. Method: Leveraging over 100,000 user reviews of AI products from G2.com, this study employs natural language processing, sentiment analysis, topic modeling, and multivariate regression to conduct the first large-scale quantitative assessment of how seven established HAI dimensions correlate with real-world user satisfaction. Contribution/Results: Four dimensions—adaptability, customization, fault tolerance, and security—exhibit robust, statistically significant positive associations with satisfaction; these effects are consistent across occupational groups, contradicting hypothesized role-based moderation. Systematic differences emerge between technical and non-technical users’ evaluative priorities. The study contributes: (1) identification of four high-impact HAI drivers; (2) a reproducible, scalable analytical framework for HAI-related user feedback; and (3) evidence-based design guidelines that inform feature prioritization in AI product development.

Technology Category

• Humans and AI: User Experience and Usability
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Human-AI Interaction (HAI) guidelines and design principles have become increasingly important in both industry and academia to guide the development of AI systems that align with user needs and expectations. However, large-scale empirical evidence on how HAI principles shape user satisfaction in practice remains limited. This study addresses that gap by analyzing over 100,000 user reviews of AI-related products from G2.com, a leading review platform for business software and services. Based on widely adopted industry guidelines, we identify seven core HAI dimensions and examine their coverage and sentiment within the reviews. We find that the sentiment on four HAI dimensions-adaptability, customization, error recovery, and security-is positively associated with overall user satisfaction. Moreover, we show that engagement with HAI dimensions varies by professional background: Users with technical job roles are more likely to discuss system-focused aspects, such as reliability, while non-technical users emphasize interaction-focused features like customization and feedback. Interestingly, the relationship between HAI sentiment and overall satisfaction is not moderated by job role, suggesting that once an HAI dimension has been identified by users, its effect on satisfaction is consistent across job roles.

Problem

• Examines how HAI principles impact user satisfaction empirically
• Analyzes 100K AI product reviews to identify key HAI dimensions
• Investigates role-based differences in HAI feature prioritization

Innovation

• Analyzed 100,000 AI product reviews
• Identified seven core HAI dimensions
• Linked HAI sentiment to user satisfaction