ProEvent: An Event-centric Benchmark for Proactive Agents

2026-07-20

AI Summary

This work addresses the lack of an event-centered evaluation framework in existing research on proactive agents, where open-ended tasks pose significant assessment challenges. The authors propose the first event-oriented benchmark for evaluating proactive assistance, built upon synthetic yet realistic multi-threaded, noisy, and dynamic instant messaging data. The framework introduces evaluation dimensions including response timing and correctness across single- and multi-step interactions. Leveraging large language models within a task pipeline, the system performs event detection, spatiotemporal information extraction, and contextually appropriate proactive response generation. Experiments across eight mainstream large language models reveal pervasive issues such as over-responsiveness and difficulty handling event cancellations; even the best-performing model, GPT-5.1, achieves correct behavior in only 26.7% of scenarios, underscoring both the benchmark’s rigor and its necessity for advancing proactive agent capabilities.

Abstract

Proactive agents are expected to anticipate user needs and provide autonomous assistance by perceiving environmental context without explicit instructions. A fundamental capability of such agents is to identify and track users' upcoming events, enabling continuous and event-specific assistance. For example, by recording the time and location of a planned hike, an agent can deliver weather reminders in advance or provide navigation support before departure. However, existing works on proactive agents largely overlook event-centric assistance, and the open-ended nature of proactive assistance poses challenges for reliable evaluation. To bridge these gaps, we introduce ProEvent, the first event-centric benchmark designed to assess an agent's ability to proactively maintain a user's timetable based on ongoing instant messaging chats. ProEvent provides synthesized yet realistic chats that consider the dynamic interaction among users, concurrent chat threads, and noise in the real world, and evaluates proactive agents on response timing, single-step response correctness, and multi-step response correctness. Experiments on eight LLMs and pipelines reveal that current agents frequently overact and struggle with event cancellation. Notably, even GPT-5.1 only reacts correctly in 26.7% of scenarios. Further qualitative analysis reveals fundamental limitations of current LLMs as proactive agents, particularly in detecting implicit events and reasoning from the user's first-person perspective.

Problem

• proactive agents
• event-centric assistance
• benchmark evaluation
• future event tracking
• autonomous assistance

Innovation

• proactive agents
• event-centric benchmark
• timetable maintenance
• implicit event detection
• first-person reasoning