Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents

2026-07-20

AI Summary

This work addresses the limitations of current computer-use agents (CUAs), which predominantly rely on textual feedback and struggle to support efficient progress monitoring and action traceability in multitasking scenarios. To overcome this, the authors propose Sidekick, a novel system that introduces multimodal, context-aware feedback into CUA communication design. Sidekick dynamically adapts to the user’s attention state by providing ambient cues during background operation, generating concise action summaries upon interaction resumption, and visualizing reasoning processes during foreground tasks. A user study with 30 participants demonstrates that Sidekick significantly enhances task efficiency compared to text-only or ambient-display baselines, while also improving users’ awareness of ongoing progress, error identification, and ability to rapidly reorient within complex task contexts.

Abstract

Computer Use Agents (CUAs) can autonomously execute complex, multi-step tasks within GUIs, enhancing efficiency through parallel multitasking. However, our formative studies with CUA experts and GenAI users indicated that current feedback is primarily text-based, requiring sustained attention to monitor progress and offering limited visibility to trace past GUI interactions. Based on the findings, we developed a prototype system, Sidekick, for communicating CUAs' status with multimodal feedback across different stages of interaction: (i) When CUAs run in the background, Sidekick signals its execution state through ambient cues. (ii) Upon resuming interaction with CUAs, Sidekick provides multimodal summaries of completed actions to support rapid context resumption. (iii) When CUAs operate in the foreground, Sidekick enhances transparency by verbalizing and visualizing the agent's reasoning. A study with 30 participants demonstrated that Sidekick significantly improved multitasking performance with CUAs compared to baseline systems that presented textual feedback either in a typical chat or in an ambient display. Sidekick supported progress awareness, and error and action traceability more effectively. Finally, we demonstrate the promise of Sidekick through several example applications, and discuss implications for long-horizon human-agent collaboration.

Problem

• Computer Use Agents
• multitasking
• feedback
• GUI interaction
• progress awareness

Innovation

• multimodal feedback
• computer use agents
• ambient cues
• context resumption
• agent transparency