BlueFin: Benchmarking LLM Agents on Financial Spreadsheets

2026-05-29

AI Summary

This study addresses the lack of systematic evaluation of large language models (LLMs) in synthesizing, manipulating, and understanding financial spreadsheets within authentic professional contexts. To bridge this gap, we introduce BlueFin—the first spreadsheet benchmark specifically designed for expert-level financial scenarios—comprising 131 complex tasks and 3,225 fine-grained scoring criteria. We further develop an automated evaluation pipeline featuring a hybrid LM-based adjudicator that integrates expert validation, rule-based checks, and learned metrics. Experimental results demonstrate strong alignment between the adjudicator and human experts (Krippendorff’s α = 0.826, macro-F1 = 0.839), while state-of-the-art LLMs achieve average scores below 50%, particularly struggling with dynamic correctness. The dataset and evaluation framework are publicly released.

Abstract

We present BlueFin, a benchmark that tasks large language model (LLM) agents with synthesis, manipulation, and comprehension tasks over spreadsheet workbooks in the professional finance domain. Though estimates of the global population of paying users of spreadsheet software range in the hundreds of millions -- an order of magnitude more than the estimated global population of professional developers -- comparatively fewer resources have been devoted to exploring and expanding LLM capabilities in the spreadsheet domain, with fewer still dedicated to mirroring real occupational tasks encountered by those in professional finance roles. In response, we curate a set of 131 challenging, complex tasks with real-world relevance in the domain, containing 3,225 granular rubric criteria; notably, our rubric criteria and LM judge evaluations are validated by a team of expert human annotators, resulting in high-quality, granular evaluations of complex tasks that are difficult to verify programmatically but can be reliably evaluated by an LM judge agent. Our judge achieves parity with expert consensus ($α=0.826$) with a macro-F1 score of 0.839. Frontier LLMs demonstrate poor performance on the challenging benchmark, with the strongest LLMs achieving less than 50\% average scores across tasks -- models exhibit particular weaknesses in dynamic correctness. Our contributions include a dataset of examples across three categories of spreadsheet tasks, an open source harness and agentic evaluation framework, and a characterization of existing frontier models' performance on our benchmark.

Problem

• LLM agents
• financial spreadsheets
• benchmarking
• task evaluation
• professional finance

Innovation

• spreadsheet understanding
• LLM agent benchmarking
• financial domain tasks
• LM judge evaluation
• dynamic correctness