Document Quality Scoring for Web Crawling

2025-04-15

AI Summary

The proliferation of low-quality web content severely degrades the efficiency of search engine crawling and indexing. Method: This paper proposes a semantic quality-aware dynamic crawling framework that, for the first time, integrates a neural semantic quality estimator—based on the model by Chang et al.—into crawl priority decisions. By jointly leveraging HTML structural and textual semantic features, we design a lightweight, Dockerized quality scoring service. The framework supports static index pruning, hierarchical indexing, and adaptive crawl scheduling to minimize resource consumption on low-quality pages. Contribution/Results: Experiments demonstrate that semantic-quality–guided crawling significantly improves search relevance, notably increasing NDCG@10. The system is modular, plug-and-play, and enables scalable, production-ready deployment. It establishes a novel, quality-driven paradigm for web information acquisition.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Semantic Web
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web crawling and indexing
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

The internet contains large amounts of low-quality content, yet users expect web search engines to deliver high-quality, relevant results. The abundant presence of low-quality pages can negatively impact retrieval and crawling processes by wasting resources on these documents. Therefore, search engines can greatly benefit from techniques that leverage efficient quality estimation methods to mitigate these negative impacts. Quality scoring methods for web pages are useful for many processes typical for web search systems, including static index pruning, index tiering, and crawling. Building on work by Chang et al.~cite{chang2024neural}, who proposed using neural estimators of semantic quality for static index pruning, we extend their approach and apply their neural quality scorers to assess the semantic quality of web pages in crawling prioritisation tasks. In our experimental analysis, we found that prioritising semantically high-quality pages over low-quality ones can improve downstream search effectiveness. Our software contribution consists of a Docker container that computes an effective quality score for a given web page, allowing the quality scorer to be easily included and used in other components of web search systems.

Problem

• Identifying low-quality web content for efficient crawling
• Prioritizing high-quality pages to improve search effectiveness
• Extending neural quality scoring for crawling optimization

Innovation

• Neural estimators for semantic quality scoring
• Prioritizing high-quality pages in crawling
• Docker container for easy quality score integration