NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction

2025-05-21

AI Summary

Traditional heuristic algorithms and LLM-based approaches for web record extraction lack fair, dynamic, and structure-aware evaluation. Method: This paper introduces the first DOM-structure-aware dynamic evaluation framework: (1) automatically generating XPath-labeled data from MHTML snapshots; (2) designing DOM-semantics-preserving multi-format input preprocessing—Slimmed HTML, Hierarchical JSON, and Flat JSON; and (3) proposing a structure-aware scoring mechanism that exclusively measures positional hallucination, thereby eliminating interference from textual hallucination. Contributions/Results: We release the first open-source synthetic DOM benchmark dataset; establish a standardized, reproducible, cross-method, and hallucination-robust evaluation paradigm; and empirically demonstrate that Flat JSON input achieves an F1 score of 0.9567 for LLM-based extraction—significantly outperforming other input formats.

Technology Category

• Natural Language Processing: Information Extraction
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Effective evaluation of web data record extraction methods is crucial, yet hampered by static, domain-specific benchmarks and opaque scoring practices. This makes fair comparison between traditional algorithmic techniques, which rely on structural heuristics, and Large Language Model (LLM)-based approaches, offering zero-shot extraction across diverse layouts, particularly challenging. To overcome these limitations, we introduce a concrete evaluation framework. Our framework systematically generates evaluation datasets from arbitrary MHTML snapshots, annotates XPath-based supervision labels, and employs structure-aware metrics for consistent scoring, specifically preventing text hallucination and allowing only for the assessment of positional hallucination. It also incorporates preprocessing strategies to optimize input for LLMs while preserving DOM semantics: HTML slimming, Hierarchical JSON, and Flat JSON. Additionally, we created a publicly available synthetic dataset by transforming DOM structures and modifying content. We benchmark deterministic heuristic algorithms and off-the-shelf LLMs across these multiple input formats. Our benchmarking shows that Flat JSON input enables LLMs to achieve superior extraction accuracy (F1 score of 0.9567) and minimal hallucination compared to other input formats like Slimmed HTML and Hierarchical JSON. We establish a standardized foundation for rigorous benchmarking, paving the way for the next principled advancements in web data record extraction.

Problem

• Evaluating web data extraction methods lacks fair benchmarks
• Comparing traditional algorithms and LLM approaches is challenging
• Need standardized framework to assess extraction accuracy and hallucination

Innovation

• Generates datasets from MHTML snapshots
• Uses structure-aware metrics for scoring
• Optimizes LLM input with HTML slimming