web crawling

This paper addresses the low efficiency of human-computer interaction and insufficient task automation on today’s Internet by proposing and systematically constructing Agentic Web—a next-generation Internet paradigm wherein AI agents autonomously collaborate to execute user intent. Methodologically, it introduces a tri-dimensional conceptual model spanning intelligence, interaction, and economics, pioneering novel paradigms such as “agent attention economy”; proposes an LLM-based agent communication protocol, task orchestration strategy, and distributed collaboration mechanism supporting retrieval, planning, and multi-agent coordination. Contributions include: (1) establishing the first structured theoretical framework with a clear technical architecture and developmental roadmap; (2) identifying and analyzing critical challenges—including scalability; and (3) open-sourcing a continuously updated research repository to foster an open, secure, and intelligent Agentic Web ecosystem.

Prior web agent research primarily focuses on navigation and transaction tasks, neglecting large-scale structured data extraction from complex, interactive websites. Method: We introduce WebLists—a benchmark comprising 200 real-world, enterprise-grade data extraction tasks across four commercial domains—requiring agents to jointly perform webpage navigation, dynamic configuration, and high-precision schema alignment. Existing LLMs and state-of-the-art (SOTA) agents achieve only 3% and 31% recall, respectively, revealing severe generalization limitations. To address this, we propose BardeenAgent: the first agent that synthesizes generalizable CSS selectors by leveraging HTML structural regularities; it integrates HTML structure-aware action modeling with programmable replay for executable, reusable automation. Contribution/Results: On WebLists, BardeenAgent achieves 66% recall—more than double the SOTA—and reduces per-output execution cost by 3×.

Web agents frequently fail to navigate unfamiliar websites due to insufficient environmental understanding and ineffective path planning. To address this, we propose a graph-search-driven structured exploration framework that formally models web navigation as a graph search process over reusable information—enabling cross-session knowledge sharing and scalable generation of high-quality navigation trajectories. Our method integrates graph search algorithms, tight coupling with the WebArena benchmark, fine-tuning of a 7B-language-model, and interactive, URL-grounded data collection. This yields a high-fidelity dataset comprising 10K successful trajectories and 40K interaction steps. Evaluated on WebArena, the fine-tuned model achieves a 21.7% task success rate—outperforming GPT-4o mini by 2.4 percentage points and establishing a new state-of-the-art for models of comparable scale.

This work addresses the poor robustness of AI agents in complex web tasks caused by overreliance on browser-based interaction. To this end, we propose a novel API-first interaction paradigm. Methodologically, we design a task-agnostic API-first hybrid agent that integrates LLM-driven API discovery and planning, dynamic schema parsing, and unified modeling of multi-source action spaces; we further construct a systematic evaluation framework using WebArena. Our contributions are threefold: (1) the first systematic empirical validation that APIs can serve as viable mainstream alternatives to browser-based web interaction; (2) the first general-purpose, task-agnostic hybrid agent supporting both API and browser modalities without task-specific fine-tuning; and (3) achieving a 35.8% task success rate on WebArena—surpassing pure browser-based agents by over 20.0 percentage points and establishing a new state-of-the-art for task-agnostic web agents.

Traditional information retrieval (IR) is constrained by static document matching and struggles to adapt to dynamic contexts and evolving user intents. To address this, we propose *Agentic IR*, a novel paradigm grounded in large language models (LLMs) that employs AI agents to actively model and evolve *information states* under instruction-driven guidance. We formally define “information state” as a dynamic, intent-aligned representation of knowledge—shifting the IR objective from passive document retrieval to active construction of context-aware information environments. Our framework comprises task-aware modeling, a hierarchical agent architecture, and a multi-dimensional evaluation protocol. Experiments demonstrate substantial improvements in goal completion rates and interactive adaptability for complex queries. This work establishes both theoretical foundations and practical pathways toward adaptive, decision-augmented next-generation IR systems.

Existing web agents suffer from a lack of scalable, process-level supervision data, making them ill-equipped to handle multi-hop, cross-page tasks. To address this limitation, this work proposes the GTA framework, which constructs long-horizon web tasks with executable trajectories through web crawling, retrieval-guided task seed generation, in-context reasoning, and automated verification. By decoupling the task generation pipeline, GTA enables efficient and systematic construction of supervision data at scale. The authors release a large-scale, dynamic benchmark spanning over 50 multilingual websites across domains such as e-commerce, government services, forums, and news. Experimental results reveal a substantial performance gap between humans and current agents, while also facilitating fine-grained diagnostics and reproducible evaluation.

This study investigates whether structured semantic metadata—such as schema.org—remains essential for intelligent agents to achieve reliable and executable data retrieval in the era of large language models (LLMs). By constructing an LLM-as-a-judge evaluation framework, the authors systematically compare semantic-aware agents leveraging such metadata against baseline agents relying solely on open web content, assessing their performance under the FAIR (Findable, Accessible, Interoperable, Reusable) principles. Experimental results demonstrate that semantic agents achieve a 65.7% improvement in overall precision for retrieving FAIR-compliant datasets and a 46.6% gain in identifying results with machine-readable download links. This work provides the first quantitative evidence of the “last-mile utility” of the semantic web ecosystem for executable tasks, underscoring the enduring value of structured metadata even in the age of LLMs.

This work proposes an intelligent web crawling approach based on multimodal large language models (MLLMs) to overcome the limitations of traditional crawlers, which struggle with dynamic, interactive websites and rely heavily on static HTML parsing and manual customization. The method integrates a specialized toolchain for web interaction and data extraction with a structured five-stage prompting mechanism, enabling fully automated, structured data collection from “index–content” architecture websites. By deeply coupling MLLMs with purpose-built tools, the system autonomously navigates complex user interfaces without human intervention. Experimental results demonstrate that the proposed approach significantly outperforms the Anthropic Computer Use baseline across six news websites and exhibits strong generalization capabilities in e-commerce scenarios.

This work proposes the first end-to-end autonomous research system that is haltable, auditable, and supports human–AI collaboration, aiming to automate the entire scientific workflow from topic selection to manuscript writing. Built upon large language models, the system employs a multi-agent architecture integrated with a unified memory mechanism, open academic indexing for verification, executable code generation, and traceable result provenance. It further incorporates a preregistered outcome contract to enforce an evidence-based closed loop. Researchers can intervene and refine the process at any stage. Experimental results demonstrate that the system can independently or collaboratively produce high-quality, verifiable research papers that adhere to publication standards.

Existing academic data systems struggle to uniformly support diverse query types—such as retrieval, knowledge discovery, and generation—and lack interpretable execution mechanisms. This work proposes an intelligent data management system tailored for academic corpora, which automatically compiles natural language queries into interpretable directed acyclic graph (DAG) execution plans. The system integrates structure-aware knowledge representation, large language model–driven hybrid query planning, and a unified execution framework based on composable operators. By synergistically combining structured knowledge management, agent-based planning, and explainable execution, the approach supports the full spectrum of academic queries and significantly outperforms existing systems in effectiveness, efficiency, and interpretability, thereby establishing a practical foundation for agent-driven academic data management.