Hallucinations and Truth: A Comprehensive Accuracy Evaluation of RAG, LoRA and DoRA

2025-02-14

AI Summary

Selecting optimal generative AI adaptation paradigms for high-precision vertical-domain FAQ answering—particularly in healthcare, finance, and law—remains challenging due to trade-offs among factual accuracy, answer relevance, and inference latency. Method: We conduct the first large-scale empirical comparison of Retrieval-Augmented Generation (RAG), Low-Rank Adaptation (LoRA), and recently proposed Decomposed Rank Adaptation (DoRA) across 20,000 real user queries and a 400K-entry domain-specific knowledge base. Contribution/Results: DoRA emerges as a new Pareto-optimal paradigm, achieving 90.1% factual accuracy and 0.88 relevance score while maintaining the lowest latency (110 ms/query)—outperforming both RAG and LoRA. We further characterize their distinct operational boundaries along three axes: knowledge grounding capability, deployment cost, and real-time adaptability. This study provides empirically grounded guidance for production-grade generative AI selection in safety-critical domains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in Generative AI have significantly improved the efficiency and adaptability of natural language processing (NLP) systems, particularly through Retrieval-Augmented Generation (RAG), Low-Rank Adaptation (LoRA), and Weight-Decomposed Low-Rank Adaptation (DoRA). RAG integrates external knowledge to enhance factual consistency in generative outputs, while LoRA enables parameter-efficient fine-tuning of large language models (LLMs). DoRA further refines this process by optimizing fine-tuning through adaptive parameter ranking and domain-aware weight adjustments, improving learning efficiency while maintaining inference performance. This paper presents a large-scale empirical evaluation of RAG, LoRA, and DoRA, with model fine-tuning and generation performance assessed on 20,000 FAQ-based queries, while the knowledge base spans 400,000 entries. The study analyzes key performance metrics such as accuracy, relevance, and inference latency. Experimental results demonstrate that DoRA achieves the highest accuracy (90.1%), relevance score (0.88), and lowest latency (110 ms per query), outperforming both LoRA and RAG in real-world, domain-specific generative AI applications. Furthermore, this study examines the trade-offs between fine-tuning efficiency, computational cost, and real-time adaptability across different models. Findings highlight RAG's effectiveness in knowledge grounding, LoRA's cost-efficient domain adaptation, and DoRA's ability to balance fine-tuning efficiency with model precision. These insights provide practical guidance for deploying AI-driven generative systems in accuracy-critical domains such as healthcare, finance, and legal services, ensuring scalability, reliability, and optimal performance in dynamic environments.

Problem

• Evaluates RAG, LoRA, DoRA accuracy
• Assesses model fine-tuning efficiency
• Examines computational cost and adaptability

Innovation

• RAG integrates external knowledge
• LoRA enables efficient fine-tuning
• DoRA optimizes adaptive parameter ranking