ESF-Bench: Benchmarking Challenging Slot-Filling Scenarios for Real-World Enterprise Applications

2026-07-25

AI Summary

This work addresses the limitations of current large language models in enterprise-grade complex slot-filling tasks, where challenges such as multi-turn dialogues, system constraints, and anomalous user behaviors remain poorly handled. The study systematically identifies 57 core difficulty categories observed in real-world deployments and introduces ESF-Bench, a new benchmark for enterprise slot filling encompassing eight domains, 810 multi-turn dialogues, and 6,530 annotated slots. Data quality is ensured through human validation and scenario reconstruction. Evaluation on this benchmark reveals a significant performance gap: even the strongest open-source model, GPT-OSS-120b, achieves only a 20.7% slot extraction success rate. The dataset, taxonomy of difficulties, and evaluation toolkit are publicly released to advance research in structured information extraction.

Abstract

The rapid rise of large language models (LLMs) has driven transformative adoption across enterprises. However, deploying these models in real-world settings presents unique challenges due to complex system constraints and unexpected user behaviors. Among these applications, slot filling is essential for converting unstructured input into structured, actionable data. In this work, we introduce ESF-Bench, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots over 8 unique domains. Curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments, ESF-Bench exposes notable limitations in current state-of-the-art LLMs, with GPT-OSS-120b low successfully extracting slots for only 20.7% of benchmark samples. To support continued research in this area, we publicly release the benchmark dataset, taxonomy, and accompanying evaluation code on GitHub.

Problem

• slot filling
• enterprise applications
• large language models
• benchmark
• real-world deployment

Innovation

• slot filling
• enterprise applications
• benchmark
• large language models
• multi-turn dialogue