Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation

2026-09-24

AI Summary

This study addresses the hallucination and lack of authoritative grounding exhibited by small language models in financial compliance question answering. To this end, we propose a domain-adaptive retrieval-augmented generation (RAG) framework. Methodologically, a three-stage retriever integrating LegalBERT with BM25 is designed, while a compact generative model is efficiently fine-tuned via a RAFT-LoRA strategy combined with 4-bit quantization. Experimental results demonstrate that the proposed pipeline improves Recall@10 to 0.774, significantly outperforming mainstream baselines and effectively enhancing both retrieval precision and factual consistency for smaller models. Current limitations include restricted cross-domain transferability and the need for more robust groundedness evaluation protocols.

Abstract

Financial institutions operate under dense, frequently amended rulebooks, and answering a compliance question correctly requires not only fluency but verifiable grounding in the authoritative text. Large language models are attractive for this task, yet the models that firms can realistically deploy on-premise are compact ones, and compact models hallucinate obligations. We study whether a carefully domain-adapted retrieval-augmented generation pipeline closes that gap. Our retriever is built in three stages on top of LegalBERT: entailment tuning that recasts question--passage matching as premise--hypothesis reconstruction, contrastive tuning with in-batch negatives, and score-level fusion with BM25. Our generator is a compact model (2B--12B parameters) served under 4-bit quantization, either prompted or adapted with retrieval-aware fine-tuning (RAFT) through LoRA. On ObliQA, a question-answering benchmark built from the Abu Dhabi Global Market rulebooks, the staged retriever raises Recall@10 from 0.256 to 0.774 and outperforms BM25 (0.678) and E5-large-v2 (0.758), the strongest general-purpose dense encoder we tested. RAFT-LoRA then improves the composite RePASs answer-quality score for every model we could adapt, with the largest gain on the weakest one. However, the adapted models do not transfer to Australian case-law questions, and a closed-book model that receives no passages at all scores within 0.011 RePASs of the full pipeline while producing answers that cite nothing and misstate obligations. The retrieval gain is therefore measured directly, the generation gain is a gain in RePASs rather than demonstrated grounding, and grounding itself requires an evaluation protocol that RePASs does not provide.

Problem

• Regulatory Compliance
• Question Answering
• Hallucination
• Retrieval-Augmented Generation
• Financial Services

Innovation

• Retrieval-Augmented Generation
• Domain Adaptation
• LegalBERT
• RAFT-LoRA
• Regulatory Compliance QA