From Reviewers' Lens: Understanding Bug Bounty Report Invalid Reasons with LLMs

2025-11-23

AI Summary

This study addresses the pervasive problem of low-quality vulnerability reports and high reviewer burden in bug bounty programs, primarily caused by the misclassification of numerous submissions as “invalid.” To tackle this, we propose the first fine-grained taxonomy of invalidity reasons specifically for information disclosure vulnerabilities. We further introduce a novel detection framework integrating retrieval-augmented generation (RAG) with multi-model collaboration—leveraging GPT-5, DeepSeek, and a fine-tuned RoBERTa model. Evaluated on nearly 10,000 real-world reports, our approach achieves a +12.7% improvement in classification accuracy for invalid reports and significantly reduces false positive rates (FPR ↓34.2%), mitigating over-acceptance bias. Moreover, we present the first empirical evidence of systematic bias introduced by reporter reputation on triage outcomes, thereby enhancing consistency and transparency in vulnerability assessment. Our work delivers an interpretable, deployable technical pathway to improve both report quality and fairness in vulnerability evaluation.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Bug bounty platforms (e.g., HackerOne, BugCrowd) leverage crowd-sourced vulnerability discovery to improve continuous coverage, reduce the cost of discovery, and serve as an integral complement to internal red teams. With the rise of AI-generated bug reports, little work exists to help bug hunters understand why these reports are labeled as invalid. To improve report quality and reduce reviewers' burden, it is critical to predict invalid reports and interpret invalid reasons. In this work, we conduct an empirical study with the purpose of helping bug hunters understand the validity of reports. We collect a dataset of 9,942 disclosed bug bounty reports, including 1,400 invalid reports, and evaluate whether state-of-the-art large language models can identify invalid reports. While models such as GPT-5, DeepSeek, and a fine-tuned RoBERTa achieve strong overall accuracy, they consistently struggle to detect invalid cases, showing a tendency to over-accept reports. To improve invalidity detection, we build a taxonomy of rejection reasons for Information Disclosure vulnerabilities and incorporate it into a retrieval-augmented generation (RAG) framework. This approach substantially improves classification consistency and reduces bias. We also examine whether reviewer decisions may be influenced by factors beyond the content of the report. Our analysis shows that reporters with higher reputations tend to receive more favorable outcomes in borderline cases, suggesting that perceived expertise can influence review judgments. Overall, our findings highlight the challenges of invalid report identification and show that combining LLMs with structured reviewer knowledge can support more transparent and consistent vulnerability report review.

Problem

• Predicting invalid bug bounty reports to reduce reviewer workload
• Understanding rejection reasons for AI-generated vulnerability reports
• Detecting bias in reviewer decisions based on reporter reputation

Innovation

• Using large language models to identify invalid bug reports
• Building taxonomy of rejection reasons for Information Disclosure vulnerabilities
• Incorporating taxonomy into retrieval-augmented generation framework