From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference

2026-09-21

AI Summary

本文通过聚合基于转换器的语言模型对单篇文章的可靠性预测，解决了新闻来源可靠性推断的问题，提高了从文章到发布者层面的准确性。

Abstract

Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standards at source. When this process is translated into a computational approach, the problem is often formulated at the level of individual articles, with models being trained on a set of pre-labelled articles and their performance being evaluated in a test phase. In this work, we investigate news source reliability inference as a source-level prediction problem. We propose a two-stage framework in which transformer-based language models first estimate the reliability of individual articles and subsequently aggregate article-level predictions to infer the reliability of previously unseen publishers. To approximate realistic deployment conditions, we enforce a strict publisher-disjoint evaluation protocol, ensuring that no publisher appears in both training and test sets. Experiments on 19,476 political news articles from 439 English-language publishers labeled with NewsGuard reliability ratings show that aggregation substantially improves robustness and performance, increasing accuracy from approximately 0.60 at the article level to 0.69 at the publisher level. Finally, we analyze how prediction errors vary across political orientations, revealing statistically significant associations between political leaning and misclassification patterns. Overall, our findings show that publisher reliability can be inferred from aggregated textual signals alone, supporting scalable and content-based approaches to automated news source assessment.

Problem

• news source reliability
• article-level predictions
• publisher-level inference
• aggregation
• transformer-based language models

Innovation

• two-stage framework
• transformer-based language models
• article-level predictions
• publisher reliability inference
• aggregation