CLI-Anything: Towards Agent-Native Computer Use

2026-06-02

AI Summary

Traditional GUI agents rely on pixel-level visual interaction, rendering them highly sensitive to interface changes, brittle in execution, and tightly coupled to temporal sequences—thus failing to leverage AI’s strengths in structured processing and programmatic control. This work proposes an “agent-native” interaction paradigm that abandons human-like visual mimicry in favor of refactoring existing software into command-line interfaces. By employing structured commands, explicit state representations, deterministic feedback, and machine-readable protocols, this approach aligns human-computer interaction with the inherent capabilities of AI systems. The authors implement and evaluate this paradigm through CLI-Hub, a platform demonstrating substantial improvements in stability, generalization, and execution efficiency, thereby offering a novel pathway toward AI-driven software automation.

Abstract

As large language models advance in reasoning and tool use capabilities, researchers increasingly seek to leverage them for computer use agents that can interact with existing software. The dominant approach develops GUI agents that control applications through visual interfaces: interpreting screenshots, locating UI elements, and executing mouse clicks to mimic human interaction. This GUI-centric paradigm fundamentally misaligns with agent capabilities. Current GUI agents struggle with brittle pixel-level interactions, timing dependencies, and coordinate-based actions that break with interface changes. They force agents to emulate human perceptual limitations rather than leverage their computational strengths in structured data processing and programmatic control. CLI-Anything argues for agent-native computer use design. Instead of forcing agents to navigate visual layouts, we create interfaces aligned with how agents naturally operate: through structured commands, explicit state representations, and deterministic feedback. We transform existing applications into command-line harnesses that preserve functionality while exposing machine-readable protocols optimized for AI-native interaction. This eliminates the lossy visual-to-computational translation that plagues GUI agents. Rather than building sophisticated screen readers and click simulators, we should redesign interaction paradigms around agent strengths: precise programmatic control and deterministic execution. We examine the methodology, architecture, evidence, and future directions for this agent-native transformation of computer use. We have built CLI-Hub as a comprehensive platform that operationalizes this agent-native computer use vision. The platform provides methodology, architecture, and infrastructure for this fundamental transformation of computer use.

Problem

• GUI agents
• agent-native interaction
• computer use
• CLI
• programmatic control

Innovation

• agent-native
• command-line interface
• programmatic control
• deterministic execution
• CLI-Hub