Problem
Research questions and friction points this paper is trying to address.
computer-use agents
GUI benchmarking
macOS
cross-platform evaluation
Apple Silicon
MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
🤖 AI Summary
Current agent evaluation benchmarks lack coverage of the macOS platform, particularly Apple Silicon architecture and native GUI tasks. This work proposes MacArena—the first large-scale online macOS evaluation environment built upon Apple’s native Virtualization framework—which integrates ported OSWorld tasks with newly designed native tasks spanning 50 applications and 421 human-validated scenarios. Experimental results reveal that state-of-the-art models suffer an average performance drop exceeding 26% on native tasks compared to ported ones, and their relative rankings invert between task types. These findings highlight fundamental limitations in cross-platform GUI understanding and demonstrate the effectiveness of macOS as a high-difficulty benchmark environment for evaluating embodied AI agents.
📝 Abstract
Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives, and their capabilities have advanced rapidly, driven in part by standardized online evaluation benchmarks such as OSWorld, which serve both as evaluation tools and as training environments for reinforcement learning. However, macOS remains underserved in this landscape: the only existing benchmark, macOSWorld, covers a narrow slice of first-party applications with simpler tasks, and runs on x86 virtual machines incompatible with Apple Silicon. We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications that combines a curated port of OSWorld tasks, content sourced from macOSWorld, and 49 new macOS-native tasks, all running on Apple's native Virtualization framework on Apple Silicon. We argue that macOS presents distinct GUI challenges beyond what Linux-based benchmarks capture, and our evaluation supports this claim: strong model performance on existing benchmarks can reflect familiarity with task distributions rather than genuine cross-platform GUI competence. Notably, model rankings invert between ported and macOS-native tasks, with a leading model trailing by over 26% on the MacArena subset, suggesting that macOS poses a genuinely harder environment for current GUI agents.
Innovation
Methods, ideas, or system contributions that make the work stand out.
MacArena
computer-use agents
GUI benchmarking
Apple Silicon
cross-platform evaluation
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
