How public datasets constrain the development of diversity-aware news recommender systems, and what law could do about it

2025-10-07

AI Summary

Public news datasets exhibit structural deficiencies in supporting diversity-aware recommender systems—specifically lacking user-level representations of pluralistic interests, multi-dimensional content attribute annotations, and cross-platform dissemination trajectories—thus failing to fulfill democratic media functions. Method: Through systematic literature review and data requirements modeling, we first formally define the essential data elements for diversity-oriented news recommendation. We further conduct a policy assessment aligned with the EU’s Digital Services Act (DSA) and General Data Protection Regulation (GDPR), evaluating legal mechanisms—including strengthening research exceptions and establishing trusted data spaces—to ensure compliant and sustainable research data access. Contribution/Results: This work establishes a theoretical framework and institutional pathway for news recommendation systems that jointly uphold algorithmic diversity and regulatory legitimacy, advancing both technical design and governance-aligned data infrastructure for democratic digital media.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: AI & Law, Justice, Regulation & Governance

Application Category

• Responsible Web: Ethical and legal aspects of web-scale data analysis, uses and collection practices
• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

News recommender systems increasingly determine what news individuals see online. Over the past decade, researchers have extensively critiqued recommender systems that prioritise news based on user engagement. To offer an alternative, researchers have analysed how recommender systems could support the media's ability to fulfil its role in democratic society by recommending news based on editorial values, particularly diversity. However, there continues to be a large gap between normative theory on how news recommender systems should incorporate diversity, and technical literature that designs such systems. We argue that to realise diversity-aware recommender systems in practice, it is crucial to pay attention to the datasets that are needed to train modern news recommenders. We aim to make two main contributions. First, we identify the information a dataset must include to enable the development of the diversity-aware news recommender systems proposed in normative literature. Based on this analysis, we assess the limitations of currently available public datasets, and show what potential they do have to expand research into diversity-aware recommender systems. Second, we analyse why and how European law and policy can be used to provide researchers with structural access to the data they need to develop diversity-aware news recommender systems.

Problem

• Public datasets limit diversity-aware news recommender system development
• European law could enable data access for diversity-focused recommenders
• Current datasets lack editorial values needed for democratic news diversity

Innovation

• Identifies dataset requirements for diversity-aware recommenders
• Assesses limitations of current public datasets availability
• Proposes European law to enable structural data access