WebTrust: An AI-Driven Data Scoring System for Reliable Information Retrieval

2025-06-05

arXiv.org

AI Summary

Current AI tools lack quantitative assessment and explainable support for web information credibility. To address this, we propose the first end-to-end fine-grained credibility assessment system. Our method leverages a fine-tuned IBM Granite-1B model integrated with a custom-constructed credibility-annotated dataset and a prompt-engineering-driven evaluation framework, producing both a reliability score (0.1–1.0) and a natural-language explanation for each claim. The key contribution is the unified modeling of fine-grained quantification and interpretability—marking the first such approach and overcoming the limitation of mainstream search engines, which provide no explicit credibility indicators. Experimental results demonstrate superior performance over comparable small-scale models and rule-based baselines across MAE, RMSE, and R² metrics. A user study further confirms statistically significant improvements in perceived information trustworthiness (p < 0.01) and user satisfaction.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

As access to information becomes more open and widespread, people are increasingly using AI tools for assistance. However, many of these tools struggle to estimate the trustworthiness of the information. Although today's search engines include AI features, they often fail to offer clear indicators of data reliability. To address this gap, we introduce WebTrust, a system designed to simplify the process of finding and judging credible information online. Built on a fine-tuned version of IBM's Granite-1B model and trained on a custom dataset, WebTrust works by assigning a reliability score (from 0.1 to 1) to each statement it processes. In addition, it offers a clear justification for why a piece of information received that score. Evaluated using prompt engineering, WebTrust consistently achieves superior performance compared to other small-scale LLMs and rule-based approaches, outperforming them across all experiments on MAE, RMSE, and R2. User testing showed that when reliability scores are displayed alongside search results, people feel more confident and satisfied with the information they find. With its accuracy, transparency, and ease of use, WebTrust offers a practical solution to help combat misinformation and make trustworthy information more accessible to everyone.

Problem

• Evaluating credibility of online information
• Providing reliability scores for search results
• Justifying trustworthiness assessments with explanations

Innovation

• Fine-tuned Granite-1B model for reliability scoring
• Integrated search engine with continuous trust evaluation
• Combines textual explanations with numerical reliability scores