Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agent Software

2025-09-17

AI Summary

This study identifies a significant gap between industry marketing claims and real-world user experiences with AI agents. To address this, we systematically analyzed the market positioning of 102 commercial AI agents and conducted a task-based user study involving 31 participants, focusing on two leading tools—Operator and Manus—to empirically benchmark claimed capabilities against actual usability. Results reveal that while users express overall positive sentiment, they encounter severe difficulties in collaborative interaction, intent interpretation, and task adaptation—rooted primarily in misalignment between agent capabilities and users’ mental models. We introduce “metacognitive capability” as a critical design dimension for bridging this gap, defined as an agent’s capacity to monitor, explain, and adapt its reasoning and behavior in response to user expectations and contextual feedback. This conceptualization provides both theoretical grounding and actionable design principles for rethinking AI agent interaction paradigms and enhancing human–agent collaboration.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Agentic search

Abstract

There is growing imprecision about what "AI agents" are, what they can do, and how effectively they can be used by their intended users. We pose two key research questions: (i) How does the tech industry conceive of and market "AI agents"? (ii) What challenges do end-users face when attempting to use commercial AI agents for their advertised uses? We first performed a systematic review of marketed use cases for 102 commercial AI agents, finding that they fall into three umbrella categories: orchestration, creation, and insight. Next, we conducted a usability assessment where N = 31 participants attempted representative tasks for each of these categories on two popular commercial AI agent tools: Operator and Manus. We found that users were generally impressed with these agents but faced several critical usability challenges ranging from agent capabilities that were misaligned with user mental models to agents lacking the meta-cognitive abilities necessary for effective collaboration.

Problem

• Examining industry conceptions and marketing of AI agents
• Identifying user challenges with commercial AI agent usability
• Assessing misalignment between agent capabilities and user mental models

Innovation

• Systematic review of commercial AI agents
• Usability assessment with participant testing
• Identified misaligned capabilities and collaboration challenges