Open Vocabulary Domain Unlearning

2026-09-25

AI Summary

This study addresses the failure of existing domain unlearning methods for vision-language models (VLMs) to generalize to unseen categories. To this end, we propose an open-vocabulary domain unlearning protocol and a surgical parameter editing framework that achieves class-agnostic domain erasure. The method leverages Fisher information matrix masking to preserve foundational generalization capabilities, while employing preference-based Tangential Manifold Scattering (TMS) to locally disrupt stylistic geometric structures, thereby precisely eliminating target domain features. Experimental results demonstrate that the proposed framework significantly enhances open-vocabulary generalization across multiple datasets, surpassing the peak 8-shot performance of baselines using only four samples. This work establishes an efficient new paradigm for the safe editing of VLMs.

Abstract

Vision-Language Models (VLMs) exhibit remarkable zero-shot generalization, yet they often encode unwanted or hazardous stylistic domains such as idealized textbook diagrams in medical AI or cartoon vehicles in autonomous driving. Approximate Domain Unlearning (ADU) aims to selectively erase a model's recognition of a target visual domain while preserving accuracy on the remaining domains. However, existing ADU methods operate under a flawed closed-vocabulary assumption: they evaluate unlearning solely on the specific object classes seen during the unlearning fine-tuning phase. Consequently, these methods do not unlearn the domain itself; they merely overfit to seen class-domain pairs, leaving the domain easily recognizable for unseen classes and providing a false sense of removal. We argue that true domain erasure must be class-agnostic. To address this, we formalize Open-Vocabulary Domain Unlearning (OVDU), a rigorous protocol that mandates domain forgetting must transfer to held-out classes. To solve the OVDU challenge, we propose a surgical parameter-editing framework. First, a Fisher Information mask isolates domain-sensitive weights, mathematically protecting foundational zero-shot generalization. Second, our Targeted Manifold Scattering (TMS) objective uses preference-based mining to locally scatter the forget domain's stylistic geometry. Evaluated across PACS, OfficeHome, and DomainNet, our method vastly improves open-vocabulary generalization over existing baselines. Crucially, it delivers exceptional sample efficiency, outperforming peak 8-shot baseline results with only 4 shots.

Problem

• Open-Vocabulary Domain Unlearning
• Vision-Language Models
• Approximate Domain Unlearning
• Domain Erasure
• Zero-shot Generalization

Innovation

• Open-Vocabulary Domain Unlearning
• Fisher Information Mask
• Targeted Manifold Scattering
• Parameter Editing
• Vision-Language Models