OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

2026-07-29

AI Summary

This work addresses the lack of benchmarks evaluating the cost-effectiveness of large language model (LLM) agents in long-horizon office suite tasks. The authors introduce a novel benchmark comprising 100 real-world office tasks, carefully anonymized for privacy, and incorporate task-level economic signals—specifically human labor time and proxy pricing—to enable direct comparison between LLM inference costs and human labor expenses, as well as value-weighted performance assessment. To ensure evaluation reliability, they design a code validator based on fine-grained scoring rules and release the full dataset and associated tools publicly. Experimental results demonstrate that while state-of-the-art LLMs currently outperform humans in terms of speed and monetary cost, their output quality still falls short of human-level performance.

Abstract

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.

Problem

• LLM agents
• office-suite tasks
• long-horizon tasks
• economic grounding
• benchmarking

Innovation

• LLM agents
• long-horizon tasks
• economic grounding
• office-suite benchmark
• cost-quality evaluation