Can I Take Another Dose? Evaluating LLM Decision-Making Under Temporal Uncertainty in OTC Dosing QA

2026-06-02

AI Summary

This study investigates the safety decision-making capabilities of large language models (LLMs) in over-the-counter (OTC) medication scenarios involving repeated dosing, temporal uncertainty, and incomplete medication histories. To this end, we introduce DOSEBENCH, a benchmark comprising 81 carefully curated adult OTC use cases, along with the first fine-grained evaluation framework that integrates temporal reasoning, adherence to dosage constraints, and handling of uncertainty. Leveraging human-annotated gold-standard answers, we conduct a multidimensional assessment of four prominent LLMs, evaluating correctness, consistency, verifiability of explanations, and failure modes. Our findings reveal that current models struggle with 24-hour rolling dose calculations and ambiguous contexts, and critically, even high-confidence responses may violate safety constraints—highlighting fundamental deficiencies in LLMs’ capacity for medical temporal reasoning.

Abstract

Large language models (LLMs) are increasingly used for everyday health questions, including whether a user can safely take another dose of an over-the-counter (OTC) medication. Yet this common safety-relevant setting remains underexplored in existing medical QA evaluations, where correct answers require tracking dose timing, computing rolling 24-hour intake, following product-label constraints, and handling incomplete medication histories. We introduce DOSEBENCH, a focused benchmark of 81 curated OTC dosing scenarios focused on adult acetaminophen and ibuprofen use, with manually annotated gold references. We evaluate four LLMs across repeated runs using metrics for decision correctness, consistency, explanation verifiability, failure types, and confidence-related signals, resulting in 1,620 model responses. Our results show that models frequently struggle with rolling-window reasoning and ambiguity-sensitive cases and that stable or confident-looking responses can still violate dosing constraints. These findings suggest that OTC dosing QA provides a narrow yet practical testbed for evaluating temporal reasoning, constraint following, and safety-relevant uncertainty handling in medical QA.

Problem

• temporal uncertainty
• OTC dosing
• medical QA
• dose timing
• safety-relevant decision-making

Innovation

• temporal reasoning
• safety-critical QA
• dose constraint
• rolling-window reasoning
• uncertainty handling