Technology Category
Natural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)Humans and AI: Other Foundations of Human Computation & AI
Alpha Excel Benchmark
🤖 AI Summary
Existing AI benchmarks emphasize abstract academic tasks while neglecting real-world business scenarios. Method: We introduce ExcelBench, the first open-source benchmark for evaluating business-level Excel proficiency. It is built upon 113 practical problems from the Financial Modeling World Cup (FMWC), augmented with a programmatic JSON conversion pipeline and a multi-dimensional task taxonomy—covering pattern recognition, chained numerical reasoning, and more—and integrates an automated execution and verification framework. Crucially, we incorporate 150 million real-world Excel workloads from daily users as a core evaluation dimension and design a fine-grained attribution-based evaluation framework. Contribution/Results: Extensive cross-model evaluation on GPT-4, Claude, and Gemini reveals strong performance in structured pattern recognition (>78%) but severe bottlenecks in complex chained numerical reasoning (<32%). ExcelBench provides a reproducible, scalable, and quantifiable metric for assessing LLMs’ practical utility in productivity tools.
Technology Category
Application Category
📝 Abstract
This study presents a novel benchmark for evaluating Large Language Models (LLMs) using challenges derived from the Financial Modeling World Cup (FMWC) Excel competitions. We introduce a methodology for converting 113 existing FMWC challenges into programmatically evaluable JSON formats and use this dataset to compare the performance of several leading LLMs. Our findings demonstrate significant variations in performance across different challenge categories, with models showing specific strengths in pattern recognition tasks but struggling with complex numerical reasoning. The benchmark provides a standardized framework for assessing LLM capabilities in realistic business-oriented tasks rather than abstract academic problems. This research contributes to the growing field of AI benchmarking by establishing proficiency among the 1.5 billion people who daily use Microsoft Excel as a meaningful evaluation metric that bridges the gap between academic AI benchmarks and practical business applications.
Problem
Research questions and friction points this paper is trying to address.
Evaluating LLMs using real-world Excel challenges from FMWC
Comparing LLM performance in business-oriented tasks via standardized benchmarks
Assessing AI proficiency in practical Excel applications for 1.5B users
Innovation
Methods, ideas, or system contributions that make the work stand out.
Converts FMWC Excel challenges into JSON formats
Benchmarks LLMs on business-oriented Excel tasks
Links academic AI benchmarks to practical applications
🔎 Similar Papers
No similar papers found.
