Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh

2026-07-26

AI Summary

This study addresses the challenges small language models face in bilingual legal question answering, particularly their limited ability to effectively leverage retrieved legal statutes and their susceptibility to language drift from Bengali to English. To mitigate these issues, the authors propose a context-injected fine-tuning approach, training Qwen3.5 models (0.8B/2B/4B) on a curated dataset of 2,165 bilingual legal QA pairs. Experimental results demonstrate that this method substantially enhances statutory utilization—evidenced by the 0.8B model’s score increasing from 2 to 34 under the 2022 English FAISS setting—and dramatically reduces language drift rates from 44–53% to 0.2–0.7% (p<0.001). These findings confirm that fine-tuning simultaneously strengthens both cross-lingual consistency and adherence to legal provisions.

Abstract

A small language model can receive the governing statutory provision and still answer incorrectly. We test whether fine-tuning on examples containing relevant law improves later use of retrieved law. We curate 2{,}165 bilingual QA records from six Bangladeshi acts and three schedules, then fine-tune Qwen3.5 at 0.8B, 2B, and 4B. Evaluation uses the 2022 and 2023 Bangladesh Bar Council exams in Bangla and machine-translated English, with no retrieval, BM25, or FAISS, scored by strict consistency over three seeded runs. At 0.8B, fine-tuning raises the 2022 English FAISS score from 2 to 34 of 100. Gains at 0.8B and 2B survive paired testing, but the 4B model has no detectable net gain: Bangla improves while several English conditions regress. Fine-tuning also reduces answers that drift from Bangla into mostly English from 44.0--53.2\% to 0.2--0.7\%, with adjusted $p<.001$ at every scale. Retrieval quality is therefore not the only bottleneck. Small bilingual legal models also differ in how they use supplied law and whether they answer in the requested language. The dataset is publicly available at https://huggingface.co/datasets/momahadi/bangladesh-legal-qa-dataset.

Problem

• legal QA
• small language models
• bilingual legal reasoning
• law usage
• language consistency

Innovation

• Context-Injected Fine-Tuning
• Legal QA
• Small Language Models
• Bilingual Legal Dataset
• Law Utilization