Humans Cannot Detect AI-Generated Media But Communities May -- For Now: Collaborative AI Detection in r/RealOrAI on Reddit

2026-05-22

AI Summary

This study addresses the challenge that individuals struggle to accurately discern AI-generated images and lack large-scale behavioral data on detection in naturalistic settings. Leveraging one year of collaborative authenticity judgments from the Reddit community r/RealOrAI—combining bot moderation with user voting—the authors construct the first large-scale dataset of human detection behavior in a real-world context. Applying a large language model to classify 10,000 reasoning comments across six cue categories (perceptual features, context, consistency, AI knowledge, domain expertise, and source tracing), they find that individuals rely predominantly (70%) on perceptual cues and rarely (4%) on source verification. However, collective deliberation amplifies the use of source verification by 4.3×, elevating overall community accuracy to 72%, thereby demonstrating that information aggregation effectively mitigates the limitations of individual heuristic judgment.

Abstract

We study human AI-detection behaviour at scale using a year of activity from r/RealOrAI, a Reddit community where users collaboratively assess whether visual media is real or AI-generated. The community is moderated by a bot that solicits verified labels from submitters of self-challenging "[GUESS]" posts and publishes an aggregate community prediction for each post, yielding naturalistic ground truth at scale. Community detection accuracy reaches 72% on [GUESS] posts with a systematic false-positive bias that intensifies over the year as the community's AI-suspicion grows. Using a six-LLM ensemble validated against human-annotated ground truth, we classify 10k reasoning-bearing comments along six cues covering perceptual features, context, consistency, AI knowledge, subject-matter expertise and provenance (tracing the media to its source). Perceptual features (scene, visual artifacts, anatomy physics, lighting, behavior, text, audio) dominate reasoning (70%) while provenance verification is rarest (4%) at the individual level but is amplified 4.3x in community summaries, revealing aggregation as a reliability filter that selectively surfaces diagnostic evidence. These findings reveal the limits of heuristic-based detection and show how online communities collectively navigate an increasingly contested information environment.

Problem

• AI-generated media
• human detection
• collaborative detection
• online communities
• media authenticity

Innovation

• collaborative AI detection
• provenance verification
• perceptual cues
• community aggregation
• LLM ensemble reasoning