Decision-Theoretic Stopping Rules for Document Screening

2026-06-05

AI Summary

This study addresses the suboptimal decision-making in document screening, where existing stopping strategies focus solely on recall while neglecting the actual costs and benefits of review tasks. To bridge this gap, the work introduces decision theory into the problem for the first time, deriving three adaptive stopping strategies grounded in the Expected Value of Perfect Information (EVPI). These strategies are integrated within a Technology-Assisted Review (TAR) framework to align stopping decisions with task-specific utility objectives. Empirical evaluations on the CLEF-IP patent dataset and medical systematic review corpora demonstrate that the proposed approach significantly improves net utility compared to prevailing stopping rules, achieving better alignment between screening outcomes and real-world review goals.

Abstract

Deciding when to stop reviewing the results of a search is a common problem with multiple applications. Existing stopping rules developed within Technology-Assisted Review (TAR) aim to achieve a pre-specified recall target and do not take into account the reason for examining the results, potentially leading to sub-optimal recommendations. This paper applies decision theory to the problem and uses it to derive three practical stopping policies based on the Expected Value of Perfect Information. The approach is applied to two professional search tasks: patent examining and systematic reviewing. Experiments on CLEF-IP and medical systematic review datasets show that the proposed approach generally produces more appropriate stopping decisions than existing methods, as demonstrated by higher net utility under the evaluated cost and payoff settings.

Problem

• stopping rules
• document screening
• Technology-Assisted Review
• recall target
• search results

Innovation

• Decision Theory
• Stopping Rules
• Expected Value of Perfect Information
• Technology-Assisted Review
• Net Utility