MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation

2026-06-20

AI Summary

This work addresses the high computational cost and catastrophic forgetting challenges faced by pretrained language models in unsupervised domain adaptation. The authors propose a parameter-efficient adaptation strategy that unifies invertible adapters and LoRA into a single PEFT framework, optimized via a hybrid objective: minimizing classification loss on labeled source-domain data while performing masked language modeling on unlabeled target-domain data. Updating only 7% of model parameters, the method achieves an average improvement of 1.41 percentage points across 20 domain-transfer tasks from MNLI, significantly outperforming existing parameter-efficient approaches such as UDapter and full fine-tuning baselines like DANN and DSN, while effectively balancing task performance and retention of domain knowledge.

Abstract

Pre-trained language models struggle when applied to new domains, as full fine-tuning is computationally expensive and prone to catastrophic forgetting. This study addresses this challenge by presenting a novel parameter-efficient strategy for unsupervised domain adaptation that combines custom PEFT architectures with mixed-objective training. Our approach simultaneously optimizes classification performance on labeled source domain data and masked language modeling (MLM) on unlabeled target domain data, preserving target domain knowledge while adapting to source domain tasks. Our method employs a custom union of invertible adapters and Low-Rank Adaptation (LoRA) within a unified parameter-efficient framework. Through comprehensive evaluation on the Multi-Genre Natural Language Inference (MNLI) dataset across 20 domain shifts, our approach achieves significant improvements over existing methods: 1.41 percentage points over the current parameter-efficient state-of-the-art UDapter, 1.26 percentage points over the fully-tuned DANN baseline, and 0.86 percentage points over DSN, while utilizing only 7% of the model's trainable parameters. These results establish new benchmarks for parameter-efficient unsupervised domain adaptation and demonstrate that carefully designed PEFT combinations with concurrent optimization can outperform both existing parameter-efficient methods and traditional fully-tuned approaches.

Problem

• unsupervised domain adaptation
• parameter-efficient fine-tuning
• pre-trained language models
• catastrophic forgetting
• domain shift

Innovation

• MixedPEFT
• parameter-efficient fine-tuning
• unsupervised domain adaptation
• invertible adapters
• LoRA