Hyperactive Minority Alter the Stability of Community Notes

2026-02-09

AI Summary

This study investigates whether community annotation mechanisms—such as X Platform’s Community Notes—achieve genuine decentralized fact-checking or are instead dominated by a small cohort of highly active users. By analyzing all publicly available data from 2021 to 2025, reproducing the platform’s open-source consensus algorithm, and conducting counterfactual simulations, the research reveals for the first time that the system exhibits structural instability. Annotation ratings are heavily concentrated among a tiny fraction of users whose pronounced political polarization significantly shapes content visibility. Minor fluctuations in user participation can trigger substantial shifts in system outputs, indicating that epistemic authority has shifted from institutional experts to a handful of active contributors, resulting in a fragile and highly centralized power structure.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Application Domains: Humanities & Computational Social Science
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Web Mining and Content Analysis: Community question answering
• Social Networks and Social Media: Media and governance, opinion dynamics, filter bubbles, polarization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

As platforms increasingly scale down professional fact-checking, community-based alternatives are promoted as more transparent and democratic. The main substitute being proposed is community-based contextualization, most notably Community Notes on X, where users write annotations and collectively rate their helpfulness under a consensus-oriented algorithm. This shift raises a basic empirical question: to what extent do users'social dynamics affect the emergence of Community Notes? We address this question by characterizing participation and political behavior, using the full public release of notes and ratings (between 2021 and 2025). We show that contribution activity is highly concentrated: a small minority of users accounts for a disproportionate share of ratings. Crucially, these high-activity contributors are not neutral volunteers: they are selective in the content they engage with and substantially more politically polarized than the overall contributor population. We replicate the notes'emergence process by integrating the open-source implementation of the Community Notes consensus algorithm used in production. This enables us to conduct counterfactual simulations that modify the display status of notes by varying the pool of raters. Our results reveal that the system is structurally unstable: the emergence and visibility of notes often depend on the behavior of a few dozen highly active users, and even minor perturbations in their participation can lead to markedly different outcomes. In sum, rather than decentralizing epistemic authority, community-based fact-checking on X reconfigures it, concentrating substantial power in the hands of a small, polarized group of highly active contributors.

Problem

• Community Notes
• fact-checking
• user polarization
• social dynamics
• algorithmic stability

Innovation

• Community Notes
• algorithmic consensus
• participation inequality
• political polarization
• counterfactual simulation