Ghost in the Transformer: Tracing LLM Lineage with SVD-Fingerprint

2025-11-09

AI Summary

To address the growing challenge of unverifiable model provenance and deceptive claims regarding the original training of large language models (LLMs), this paper proposes GhostSpec—a lightweight, non-intrusive, data-free model fingerprinting method. GhostSpec leverages singular value decomposition (SVD) of Transformer attention weight matrices to extract structured matrix invariants that remain robust under fine-tuning, pruning, and adversarial perturbations, thereby constructing compact and resilient “provenance fingerprints.” It is the first approach to achieve fully data-free model attribution, enabling zero-shot identification of derivative relationships. Experiments demonstrate that GhostSpec maintains over 98% matching accuracy across diverse model modifications and attacks, with computational overhead under 0.1 GPU-hour. This significantly advances intellectual property protection for LLMs and enhances trustworthiness in open-source model ecosystems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Large Language Models (LLMs) have rapidly advanced and are widely adopted across diverse fields. Due to the substantial computational cost and data requirements of training from scratch, many developers choose to fine-tune or modify existing open-source models. While most adhere to open-source licenses, some falsely claim original training despite clear derivation from public models. This raises pressing concerns about intellectual property protection and highlights the need for reliable methods to verify model provenance. In this paper, we propose GhostSpec, a lightweight yet effective method for verifying LLM lineage without access to training data or modification of model behavior. Our approach constructs compact and robust fingerprints by applying singular value decomposition (SVD) to invariant products of internal attention weight matrices, effectively capturing the structural identity of a model. Unlike watermarking or output-based methods, GhostSpec is fully data-free, non-invasive, and computationally efficient. It demonstrates strong robustness to sequential fine-tuning, pruning, block expansion, and even adversarial transformations. Extensive experiments show that GhostSpec can reliably trace the lineage of transformed models with minimal overhead. By offering a practical solution for model verification and reuse tracking, our method contributes to the protection of intellectual property and fosters a transparent, trustworthy ecosystem for large-scale language models.

Problem

• Identifying unauthorized derivations from open-source LLMs through lineage verification
• Developing data-free non-invasive methods to trace model provenance
• Protecting intellectual property against false claims of original training

Innovation

• Uses SVD on attention weight matrices for fingerprints
• Creates data-free non-invasive model lineage verification
• Provides robust tracing against fine-tuning and transformations