web archiving

This study addresses the challenge of accurately estimating the absolute coverage of a web crawler over the crawlable URL space in the absence of external ground-truth data. The authors propose a statistical method that relies solely on longitudinal crawl data from a single crawler. By analyzing the intersections of URLs across multiple consecutive crawls, they formulate an urn-model-based estimation framework and employ linear regression to infer the coverage ratio. Notably, the approach requires neither external benchmarks nor comparisons across multiple crawlers, making it applicable to any focused longitudinal crawling scenario. Experiments on 15 semi-annual crawls of the German academic web from 2013 to 2021 demonstrate that, under stable configurations, the crawler achieves approximately 46% coverage, thereby validating the method’s effectiveness and practical utility.

This study addresses a key limitation in traditional web crawling analysis, which typically assumes URLs are uniformly distributed and thereby overlooks the heterogeneous “persistent core–dynamic periphery” structure inherent in real-world web graphs. To overcome this assumption, the authors propose a novel two-component urn model grounded in discovery curves and sliding windows, enabling the first quantitative estimation of both the core proportion and the dynamic evolution parameters of the periphery within a web corpus. By jointly fitting coverage and survival rates, the model demonstrates strong empirical validity on both Common Crawl and German academic web datasets. Results reveal a pronounced core–periphery organization in both collections, with further heterogeneity observed even within the dynamic periphery itself.

Web page longevity and archival sustainability remain poorly understood due to the lack of large-scale, temporally representative empirical studies. Method: Leveraging 26 years of Internet Archive data, we construct the first longitudinal web sample—comprising 27.3 million URLs—with strong temporal representativeness. We propose a multidimensional stratified sampling strategy based on first archival time, MIME type, URL depth, and top-level domain; integrate ZipNum indexing with the CDX API for efficient systematic sampling; and apply upsampling and downsampling to correct temporal distribution skew and domain-level bias. After filtering HTML pages and normalizing top-level URLs, we generate TimeMaps covering 3.8 billion archival records, enabling annual, million-scale balanced sampling. Contribution/Results: This work establishes the most comprehensive, reproducible empirical foundation to date for studying web persistence, enabling rigorous, longitudinal analysis of webpage survival and archival coverage.

This paper addresses the fairness and timeliness challenges in web cache refresh under bandwidth constraints, proposing a robust crawling scheduling framework leveraging noisy change signals (e.g., sitemaps, CDN notifications). The core problem lies in signal unreliability—characterized by false positives and false negatives—and heterogeneous noise levels across pages, which hinder conventional approaches from simultaneously achieving fairness and efficiency. Methodologically, we introduce the first optimal robust fusion of heterogeneous noise-side information, integrated with Poisson-process-based stochastic scheduling, distributed online rate control, and dynamic bandwidth adaptation. The system supports decentralized deployment while maintaining constant bandwidth consumption. Experiments demonstrate a 37% improvement in bandwidth utilization over signal-agnostic baselines and state-of-the-art heuristics, elimination of request latency spikes, and significant gains in cross-page scheduling fairness and temporal consistency.

This work addresses the challenge of efficiently harvesting large-scale, high-quality statistical datasets from unstructured websites. The authors propose a reinforcement learning–based focused crawling approach that formulates web page retrieval as a sleeping bandits problem, enabling dynamic learning of high-value hyperlink paths. Their proposed SB-CLASSIFIER leverages contextual features along navigation paths to intelligently identify hyperlinks leading to multi-target resource pages. Experimental evaluation on real-world websites encompassing millions of pages demonstrates that the method captures the vast majority of target resources while crawling only a small fraction of the site, significantly outperforming existing approaches in both efficiency and scalability.

Existing research on dark web content predominantly relies on static snapshots, which fail to capture the dynamic evolution of cybercriminal topics. This work proposes the first longitudinal topic modeling framework tailored for the dark web, integrating domain-specific embeddings, density-based clustering, and time-series aggregation to enable site-level topic tracking across over 11.4 million HTML snapshots collected over six years. The approach identifies 55 distinct topic clusters, revealing that 75% of discussions concentrate around a small set of persistent core themes. With a median topic lifespan of 75 months, the findings demonstrate that dark web content exhibits pronounced continuity and structured evolutionary patterns, thereby overcoming the limitations inherent in conventional static analyses.

This study identifies and systematically categorizes pervasive temporal leakage mechanisms in web-based retrieval that critically undermine the validity of retroactive forecasting evaluations. Focusing on widely used search engine date filters (e.g., Google’s “before:” operator), the authors demonstrate that such methods introduce severe temporal contamination, leading to inflated performance estimates. Through a combination of manual auditing and automated analysis using a large language model (gpt-oss-120b), they find that 71% of retrieved documents exhibit some form of temporal leakage, with 41% directly revealing the answer. Consequently, the inclusion of these leaked documents artificially reduces the Brier score from 0.242 to 0.108, substantially distorting assessment reliability and exposing fundamental flaws in date-filtered retroactive evaluation protocols.

Existing web agents suffer significant performance degradation when confronted with the temporal evolution of web interfaces, primarily due to the absence of robust evaluation frameworks and training mechanisms that account for cross-era UI variations. To address this gap, this work introduces TimeWarp, a benchmark that leverages containerization to reconstruct six distinct historical UI versions across three websites. Furthermore, we propose TimeTraj, an algorithm that employs multi-version plan distillation to generate robust action trajectories, replacing conventional single-version behavioral cloning. Experimental results demonstrate that our approach substantially enhances cross-UI generalization: success rates improve from 20.4% to 37.7% for Qwen-3 4B and from 0% to 27.0% for Llama-3.1 8B, marking the first systematic effort to enable robust evaluation and enhancement of web agents across evolving historical interfaces.

This work proposes an intelligent web crawling approach based on multimodal large language models (MLLMs) to overcome the limitations of traditional crawlers, which struggle with dynamic, interactive websites and rely heavily on static HTML parsing and manual customization. The method integrates a specialized toolchain for web interaction and data extraction with a structured five-stage prompting mechanism, enabling fully automated, structured data collection from “index–content” architecture websites. By deeply coupling MLLMs with purpose-built tools, the system autonomously navigates complex user interfaces without human intervention. Experimental results demonstrate that the proposed approach significantly outperforms the Anthropic Computer Use baseline across six news websites and exhibits strong generalization capabilities in e-commerce scenarios.

This work addresses the performance degradation of classification models in real-world deployment caused by non-stationary data distributions. It reframes non-stationary classification as a time-series forecasting problem and introduces an end-to-end trainable discrete retrieval mechanism. This mechanism employs input-dependent dynamic queries to selectively retrieve relevant information from labeled samples collected after the initial training period, and integrates a score-based gradient estimator for efficient optimization. By doing so, the method overcomes the conventional limitation that prevents models from leveraging newly annotated post-training data. Empirical results demonstrate substantial improvements in robustness to distribution shifts on both synthetic benchmarks and the Amazon Reviews '23 dataset (Electronics category), with memory consumption scaling predictably with the length of the historical sequence.