analyze webpage content

This study addresses the limitations of existing benchmarks for main content extraction from web pages, which are typically small-scale, homogeneous, and outdated, thereby failing to adequately evaluate system generalization across diverse page structures. To overcome this, the authors introduce WCXB, a new benchmark comprising 2,008 pages from 1,613 domains spanning seven structurally distinct categories, including news articles, forums, and product pages. High-quality annotations are ensured through a rigorous five-stage pipeline combining LLM-assisted labeling, automated validation, four rounds of model-based review, and human verification. Evaluation of 13 state-of-the-art extraction systems reveals that while top-performing methods achieve an F1 score of 0.93 on article pages, their performance drops substantially on non-news structured pages (F1 = 0.41–0.84), highlighting a critical generalization gap. The dataset is publicly released.

This paper addresses layout-rich document information extraction (IE), systematically exploring the LLM-driven, layout-aware IE design space. It tackles three core challenges—data structuring, model interaction, and output refinement—by proposing layout-aware prompt engineering, document chunking, and multimodal input representation. The work formally defines the complete design space for layout-aware IE for the first time and demonstrates that general-purpose LLMs, when appropriately configured, achieve performance on par with specialized models. An efficient One-Factor-at-a-Time (OFAT) tuning strategy is introduced, yielding a +14.1 F1-point gain on the LayoutLMv3 benchmark—nearly matching the +15.1 gain of exhaustive full-factorial tuning—while substantially reducing token consumption. Additionally, the authors introduce and open-source LayIE-LLM, the first comprehensive evaluation suite specifically designed for layout-aware IE with LLMs.

This work proposes an intelligent web crawling approach based on multimodal large language models (MLLMs) to overcome the limitations of traditional crawlers, which struggle with dynamic, interactive websites and rely heavily on static HTML parsing and manual customization. The method integrates a specialized toolchain for web interaction and data extraction with a structured five-stage prompting mechanism, enabling fully automated, structured data collection from “index–content” architecture websites. By deeply coupling MLLMs with purpose-built tools, the system autonomously navigates complex user interfaces without human intervention. Experimental results demonstrate that the proposed approach significantly outperforms the Anthropic Computer Use baseline across six news websites and exhibits strong generalization capabilities in e-commerce scenarios.

This work addresses high-accuracy conversion of unstructured/semi-structured documents (e.g., contracts, academic papers, invoices) into structured, machine-readable data. Method: We systematically survey and empirically compare modular pipeline approaches against end-to-end multimodal large models, proposing a unified framework integrating OCR, layout analysis (LayoutParser), graph neural networks, vision-language models (VLMs), and specialized formula/table recognition. We identify and characterize core bottlenecks—layout understanding, dense text recognition, and cross-modal alignment—for the first time. Contribution/Results: We establish a comprehensive analytical framework covering methodology, challenges, and benchmarks, revealing >32% performance gaps of current SOTA on complex layouts (e.g., multi-column, nested tables). We propose a “dual-driven” evolution path emphasizing both data diversity and scale, and open-source a larger annotated dataset to significantly advance knowledge base construction and training-data generation for large models.

This study addresses the absence of an effective evaluation framework for structured generative search summaries—comprising overviews, titled sections, and cited source documents—that appear at the top of natural search results. It presents the first systematic effort to construct a comprehensive evaluation framework tailored to these summaries, explicitly defining their core components and multidimensional assessment criteria. By integrating large language model generation techniques with established information retrieval evaluation methodologies, the work proposes a practical and scalable evaluation framework and outlines a clear empirical validation pathway. This contribution establishes a foundational methodological basis for future research on generative search summaries and their impact on user experience and information access.

Existing web content extraction methods suffer from three key limitations at scale: low efficiency (high latency of generative models), poor adaptability (weak generalization of rule-based approaches), and structural neglect (semantic loss in HTML due to chunking and re-ranking). To address these, this paper proposes a novel indexing-based paradigm that reformulates content extraction as a structure-aware discriminative index prediction task—shifting from generation to precise localization. Our method introduces an HTML-structure-aware segmentation mechanism and an addressable fragment indexing scheme, enabling lightweight discriminative models to directly predict the positions of query-relevant fragments. This fully decouples extraction latency from webpage length. To our knowledge, this is the first work to achieve such a paradigm shift. Extensive experiments demonstrate state-of-the-art performance across three tasks—RAG-based QA, main-content extraction, and query-relevant extraction—delivering higher accuracy (improved match rates), lower latency (faster inference), and stronger robustness.

This study addresses the challenges of losing diverse perspectives and lacking traceability in the analysis of large-scale heterogeneous textual corpora. To this end, it proposes a structured reading approach grounded in large language models, which defers irreversible information compression by sequentially performing insight extraction, semantic clustering, theme generation, and iterative omission detection. This pipeline explicitly preserves divergent viewpoints, thereby enhancing both coverage and auditability of the analytical process. Evaluated on a corpus of 152 industrial policy documents, the method successfully extracted over 17,500 structured insights and constructed a comprehensive thematic map. The implementation has been open-sourced as the first end-to-end framework supporting large-scale qualitative synthesis.