Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledge

2026-08-03

AI Summary

This work addresses the challenge of catastrophic forgetting when injecting proprietary knowledge into enterprise question-answering models. The authors propose a three-stage adaptation pipeline: first, constructing task-specific supervision signals from domain documents; second, performing supervised fine-tuning (SFT) augmented with replay of general-domain data; and third, introducing a novel residual error–driven reinforcement learning (RL) strategy that selectively optimizes on erroneous samples. Evaluated on WnuanBench, the approach increases the acceptable response rate from 52.76% to 91.51%, with residual-based sampling outperforming random and full-pool sampling by 3.11 and 2.97 percentage points, respectively. While general capabilities decline modestly by an average of 5.17 points—primarily affecting instruction following—the method achieves a controllable trade-off between domain performance gains and generalization loss, validated through both automated metrics and expert assessment.

Abstract

Enterprise question answering requires models to acquire proprietary knowledge without discarding general capabilities. We present Wnuan, a three-stage pipeline that constructs task-oriented supervision from documents, performs supervised fine-tuning with general-data replay, and applies reinforcement learning to residual errors. On the 707-question WnuanBench, the primary 32B route raises acceptable-answer rate (AAR) from 52.76% before adaptation to 80.06% after SFT and 91.51% after RL. Under a matched 100-update protocol, residual-error sampling outperforms full-pool and size-matched random sampling by 3.11 and 2.97 points, respectively. Source-cluster bootstrap intervals remain above zero for both contrasts, and a same-domain validation set preserves the ordering. The general-benchmark average decreases by 5.17 points across the route, concentrated in instruction following. The automatic evaluation ensemble agrees with an authoritative domain expert on 90.5% of a stratified Wnuan-Inst response sample. These results characterize both the gains and the general-capability cost of staged enterprise adaptation.

Problem

• enterprise question answering
• proprietary knowledge
• general capabilities
• post-training adaptation
• capability trade-off

Innovation

• staged post-training
• proprietary knowledge adaptation
• residual-error reinforcement learning
• general-data replay
• enterprise question answering