web scraping

This study investigates whether structured semantic metadata—such as schema.org—remains essential for intelligent agents to achieve reliable and executable data retrieval in the era of large language models (LLMs). By constructing an LLM-as-a-judge evaluation framework, the authors systematically compare semantic-aware agents leveraging such metadata against baseline agents relying solely on open web content, assessing their performance under the FAIR (Findable, Accessible, Interoperable, Reusable) principles. Experimental results demonstrate that semantic agents achieve a 65.7% improvement in overall precision for retrieving FAIR-compliant datasets and a 46.6% gain in identifying results with machine-readable download links. This work provides the first quantitative evidence of the “last-mile utility” of the semantic web ecosystem for executable tasks, underscoring the enduring value of structured metadata even in the age of LLMs.

Existing web data collection methods face dual challenges of high labor costs and poor scalability: wrapper-based approaches suffer from weak adaptability and reproducibility, while LLM-driven solutions incur prohibitive computational and financial overhead. This paper introduces Instruct2DS—the first fully automated, natural-language-instruction-driven system for structured data extraction from open web pages. Its core innovations are: (1) a task-oriented multi-agent hypergraph communication architecture enabling efficient collaboration with minimal token consumption; (2) a hypergraph caching mechanism that significantly accelerates dynamic task response; and (3) the first cross-domain benchmark, Instruct2DS, supporting real-time data acquisition. Experiments demonstrate that Instruct2DS outperforms all state-of-the-art methods on the Instruct2DS benchmark and three public benchmarks, achieving superior performance on complex tasks such as children’s book scraping and survey paper extraction. The code and dataset are publicly released.

Web agents frequently fail to navigate unfamiliar websites due to insufficient environmental understanding and ineffective path planning. To address this, we propose a graph-search-driven structured exploration framework that formally models web navigation as a graph search process over reusable information—enabling cross-session knowledge sharing and scalable generation of high-quality navigation trajectories. Our method integrates graph search algorithms, tight coupling with the WebArena benchmark, fine-tuning of a 7B-language-model, and interactive, URL-grounded data collection. This yields a high-fidelity dataset comprising 10K successful trajectories and 40K interaction steps. Evaluated on WebArena, the fine-tuned model achieves a 21.7% task success rate—outperforming GPT-4o mini by 2.4 percentage points and establishing a new state-of-the-art for models of comparable scale.

LLM-driven web data collection suffers from hallucination, semantic misinterpretation, and failure to detect invalid information. Method: This paper proposes a model-agnostic multi-agent data governance framework featuring a novel “AI Committee” architecture—comprising specialized, collaboratively coordinated agents for verification, repair, and consistency checking—operating without fine-tuning and leveraging chain-of-thought reasoning, in-context learning, and a self-correcting feedback loop for end-to-end automation. Contribution/Results: Experiments on three real-world datasets show the framework achieves 78.7% data completeness and up to 100% precision, significantly outperforming existing baselines; it also demonstrates strong generalization across diverse LLMs. To our knowledge, this is the first work to systematically apply structured multi-agent coordination to web data quality assurance, establishing a scalable, verifiable paradigm for trustworthy web data acquisition.

Current AI tools lack quantitative assessment and explainable support for web information credibility. To address this, we propose the first end-to-end fine-grained credibility assessment system. Our method leverages a fine-tuned IBM Granite-1B model integrated with a custom-constructed credibility-annotated dataset and a prompt-engineering-driven evaluation framework, producing both a reliability score (0.1–1.0) and a natural-language explanation for each claim. The key contribution is the unified modeling of fine-grained quantification and interpretability—marking the first such approach and overcoming the limitation of mainstream search engines, which provide no explicit credibility indicators. Experimental results demonstrate superior performance over comparable small-scale models and rule-based baselines across MAE, RMSE, and R² metrics. A user study further confirms statistically significant improvements in perceived information trustworthiness (p < 0.01) and user satisfaction.

This work addresses the challenge of accurately retrieving executable capabilities from heterogeneous agent marketplaces that comprise native agents, toolkits, and reusable skills. It presents the first adaptation of general-purpose text embedding models—BGE-base, KaLM-v1.5, and EasyRec—to this setting by constructing capability profiles from publicly available metadata and applying supervised fine-tuning on the AgentSelect benchmark. The approach is evaluated for its cross-catalog generalization ability, demonstrating significant performance gains on both the MuleRun native agent benchmark and the ClawHub skill benchmark, which includes 50 skills and 1,000 queries. These results validate the effectiveness and transferability of the proposed fine-tuning strategy for capability retrieval in mixed-agent environments.

Existing retrieval evaluation metrics, such as precision and recall, struggle to assess the breadth of information coverage in retrieved results, particularly in retrieval-augmented generation (RAG) scenarios where it is critical to capture diverse key information. To address this limitation, this work introduces CoverageBench, the first cross-task, multi-domain benchmark specifically designed for evaluating information coverage. CoverageBench integrates topics, fine-grained information nuggets, relevance labels, and baseline rankings, moving beyond traditional document-level relevance paradigms. Released via Hugging Face Datasets, the benchmark enables reproducible, quantitative evaluation of the diversity and comprehensiveness of retrieved information, establishing a standardized platform for advancing research on information coverage in retrieval systems.

This work addresses the challenge faced by domain experts in manually synthesizing information from the open web to answer complex analytical questions—a process that is often inefficient and labor-intensive. To automate this, the authors propose treating the open web as a latent database and introduce a novel task paradigm termed SODIUM, along with the first benchmark for this task, SODIUM-Bench. They develop SODIUM-Agent, a multi-agent system comprising a web explorer and a cache manager, which integrates an ATP-BFS exploration algorithm with structure-aware information extraction and caching strategies to automatically construct a queryable structured database. Evaluated on SODIUM-Bench, their approach achieves an accuracy of 91.1%, nearly doubling the performance of the strongest baseline and improving upon the weakest baseline by up to 73 times.

This work addresses the challenge of efficiently and reliably discovering valid web pages and extracting accessible multimodal content—such as text, images, and videos—under a fixed interaction budget. The authors propose BaRA, an agent that combines breadth-first search for link discovery with a validity verification mechanism to filter out hallucinated or broken links. To ensure content accessibility, BaRA employs rule-based multimodal checks and incorporates a history-aware introspection module to recover from execution failures and incomplete outputs. As the first approach to integrate budget constraints with site-level multimodal data collection, BaRA consistently outperforms existing agents in both synthetic and real-world website evaluations, achieving significant improvements in valid link discovery rates and successful retrieval of downloadable multimodal content.

This work addresses the severe fragmentation of open data platforms, which significantly impedes cross-source data discovery and semantic interoperability. To overcome this challenge, the authors propose a unified framework that integrates over 7.6 million datasets through semantic extraction and normalization, a scalable topic-tag graph, multi-entity–enhanced navigation, and a provenance-aware mechanism. The resulting system establishes a context-aware and traceable paradigm for data exploration. Compared to existing platforms such as ChatPD and Google Dataset Search, the proposed approach demonstrates substantial improvements in data coverage breadth, timeliness, and provenance tracking, thereby enabling more efficient and semantically rich cross-domain data discovery.