VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations

2026-08-04

AI Summary

This study addresses the critical issue of inconsistencies between generated responses and cited sources in veterinary long-form question answering, with particular emphasis on the potential harms of high-risk factual errors. The authors propose a multi-step evaluation framework that decomposes model-generated answers into atomic claims, each assigned a harm-weighted score based on its potential clinical impact and its faithfulness to the supporting source excerpts. By integrating claim decomposition, faithfulness assessment, and risk-aware scoring, the method yields a risk-weighted overall credibility score that offers multidimensional interpretability. Experimental results on an expert-annotated dataset demonstrate strong correlation between the proposed metric and judgments by veterinary specialists, with consistent performance even when employing lightweight claim-level verification models.

Abstract

Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and veterinary medicine. However, this does not guarantee that generated claims are faithful to the provided excerpts. We present VetScore, a multi-step evaluation method for veterinary long-form question answering, designed to assess how well are generated claims supported by the provided excerpts, weighing this information by each claim's harm potential. VetScore first segments the output and decomposes it into individual claims, then scores each claim with respect to its harm potential and evaluates its faithfulness to source excerpts, and finally calculates the overall risk-adjusted score. We collect an expert-annotated meta-evaluation dataset, evaluate our approach with a range of judge models, and show that it achieves high correlations with veterinary experts even with small judge models, while offering explainability across multiple dimensions.

Problem

• fact verification
• veterinary medicine
• long-form QA
• citation faithfulness
• risk-weighted evaluation

Innovation

• risk-weighted evaluation
• fact verification
• long-form QA
• faithfulness assessment
• harm potential