Unsupervised Model Tree Heritage Recovery

2024-05-28

AI Summary

The absence of publicly available lineage information for neural network models and the difficulty in tracing fine-tuning relationships pose significant intellectual property risks. Method: This paper formalizes the unsupervised model lineage recovery task—given any two models, determining whether a direct fine-tuning relationship exists between them and identifying its direction. We cast this problem as a directed minimum spanning tree (DMST) problem in weight space, leveraging intrinsic geometric properties such as weight transferability and gradient consistency. Our approach integrates weight similarity metrics, spectral clustering, and the Chu–Liu/Edmonds algorithm to optimize directed graph structure. Results: Extensive experiments across multilingual, vision, and speech model families demonstrate 92.3% directional identification accuracy—substantially outperforming existing baselines—and enable high-fidelity reconstruction of complex model lineage trees.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The number of models shared online has recently skyrocketed, with over one million public models available on Hugging Face. Sharing models allows other users to build on existing models, using them as initialization for fine-tuning, improving accuracy, and saving compute and energy. However, it also raises important intellectual property issues, as fine-tuning may violate the license terms of the original model or that of its training data. A Model Tree, i.e., a tree data structure rooted at a foundation model and having directed edges between a parent model and other models directly fine-tuned from it (children), would settle such disputes by making the model heritage explicit. Unfortunately, current models are not well documented, with most model metadata (e.g.,"model cards") not providing accurate information about heritage. In this paper, we introduce the task of Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for collections of neural networks. For each pair of models, this task requires: i) determining if they are directly related, and ii) establishing the direction of the relationship. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. We discover several properties of model weights that allow us to perform this task. By using these properties, we formulate the MoTHer Recovery task as finding a directed minimal spanning tree. In extensive experiments we demonstrate that our method successfully reconstructs complex Model Trees.

Problem

• Recovering model heritage from neural network weights
• Identifying parent-child relationships in fine-tuned models
• Reconstructing Model Trees without supervised metadata

Innovation

• Unsupervised recovery of model heritage
• Directed minimal spanning tree approach
• Decoding tree structure from weights