SentinelBench: A Benchmark for Long-Running Monitoring Agents

2026-06-03

AI Summary

This work addresses the inefficiency of existing AI agents, which predominantly employ continuous active operation strategies and thus struggle with tasks requiring prolonged monitoring and waiting for external event triggers—often leading to excessive resource consumption and delayed responses. To tackle this limitation, we introduce SentinelBench, the first open-source evaluation benchmark specifically designed for long-duration monitoring agents. It comprises 100 dynamic tasks across 10 synthetic web environments (e.g., email, calendar, finance), leveraging scripted event replay and a browser-based agent interface to clearly differentiate between continuous action and passive monitoring behaviors. The benchmark further incorporates a trade-off evaluation metric balancing response latency and resource usage. Baseline experiments across three large language models and two agent frameworks demonstrate that architectural choices significantly impact task success rates, responsiveness, and resource efficiency.

Abstract

AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.

Problem

• long-running tasks
• monitoring agents
• sustained attention
• resource efficiency
• event-driven response

Innovation

• long-running monitoring agents
• sustained attention
• SentinelBench
• time-evolving environments
• agent resource efficiency