Unified Attacks to Large Language Model Watermarks: Spoofing and Scrubbing in Unauthorized Knowledge Distillation

2025-04-24

AI Summary

This work addresses the insufficient robustness of large language model (LLM) watermarking against unauthorized knowledge distillation—where existing attacks either require white-box access or fail to simultaneously achieve watermark removal and forgery. We propose CDG-KD, the first black-box, unified bidirectional attack framework that requires no internal model access. Leveraging a newly discovered “watermark radioactivity” phenomenon—i.e., the propagation and amplification of watermark signals during distillation—we design a contrastive decoding guidance mechanism to jointly model watermark erasure and adversarial watermark forging. Extensive experiments across multiple LLMs and watermarking schemes demonstrate: >92% watermark removal rate, >89% successful forgery rate, and <1.5% degradation in distilled models’ general-purpose performance. To our knowledge, this is the first systematic study to characterize and exploit watermark propagation dynamics in knowledge distillation, establishing a new evaluation paradigm and a strong baseline for assessing LLM watermark security.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Watermarking has emerged as a critical technique for combating misinformation and protecting intellectual property in large language models (LLMs). A recent discovery, termed watermark radioactivity, reveals that watermarks embedded in teacher models can be inherited by student models through knowledge distillation. On the positive side, this inheritance allows for the detection of unauthorized knowledge distillation by identifying watermark traces in student models. However, the robustness of watermarks against scrubbing attacks and their unforgeability in the face of spoofing attacks under unauthorized knowledge distillation remain largely unexplored. Existing watermark attack methods either assume access to model internals or fail to simultaneously support both scrubbing and spoofing attacks. In this work, we propose Contrastive Decoding-Guided Knowledge Distillation (CDG-KD), a unified framework that enables bidirectional attacks under unauthorized knowledge distillation. Our approach employs contrastive decoding to extract corrupted or amplified watermark texts via comparing outputs from the student model and weakly watermarked references, followed by bidirectional distillation to train new student models capable of watermark removal and watermark forgery, respectively. Extensive experiments show that CDG-KD effectively performs attacks while preserving the general performance of the distilled model. Our findings underscore critical need for developing watermarking schemes that are robust and unforgeable.

Problem

• Assessing watermark robustness against scrubbing attacks
• Evaluating watermark unforgeability under spoofing attacks
• Developing unified framework for bidirectional watermark attacks

Innovation

• Contrastive decoding extracts watermark texts
• Bidirectional distillation trains student models
• CDG-KD framework enables unified watermark attacks