news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling

2026-07-23

AI Summary

Accurately extracting structured content from structurally heterogeneous news webpages—often cluttered with noise such as advertisements and navigation elements—remains challenging, as rule-based methods suffer from poor generalization and large language models incur high computational costs. This work proposes the first application of a lightweight long-context language model to the news extraction task, fine-tuned on high-quality, human-verified data from the Fundus library to enable end-to-end conversion from raw HTML to both clean Markdown text and structured JSON (including fields such as title, author, date, and body). The approach achieves substantial improvements over strong baselines while maintaining low computational overhead, yielding gains of 4.8 BLEU and 6.1 METEOR on HTML-to-Markdown translation, and 2.2 BLEU and 4.1 METEOR on HTML-to-JSON generation. All models and data are publicly released.

Abstract

Extracting structured content from news pages remains challenging due to heterogeneous HTML layouts, inconsistent markup, and substantial boilerplate such as navigation elements and advertisements. Rule-based news crawlers can achieve high extraction accuracy by encoding site-specific structure, but require manual configuration in order to generalize to new publishers. Large language models provide a more flexible alternative by reducing the need for handcrafted rules, but their high computational cost limits practical deployment. In this paper, we introduce news-crawler-LM, a small long-context language model fine-tuned on high-quality, human-validated extractions from the Fundus news-crawling library. Our model converts raw HTML into plaintext and structured JSON, including fields such as headline, author, publication date, and article body. In our experiments, news-crawler-LM outperforms strong baselines in HTML-to-Markdown and HTML-to-JSON extraction, improving performance by +4.8 BLEU and +6.1 METEOR in the HTML-to-Markdown task, and by +2.2 BLEU and +4.1 METEOR in the HTML-to-JSON task. However, we also observe that our model only slightly better compared to other rule-based parsing libraries on the HTML-to-plaintext task in evaluations on previously unseen publishers. We release all models and artifacts to the research community.

Problem

• news crawling
• structured content extraction
• HTML parsing
• boilerplate removal
• web scraping

Innovation

• news crawling
• long-context language model
• structured content extraction
• HTML-to-JSON
• fine-tuned small LM