Bergson: An Open Source Library for Data Attribution

2026-06-10

AI Summary

Current research on data attribution is hindered by the lack of efficient, scalable open-source tools capable of analyzing the influence of training data on large language models. This work proposes and releases an extensible data attribution library that, for the first time, integrates three state-of-the-art algorithms—MAGIC, SOURCE, and TrackStar. By incorporating disk-based gradient storage, multi-node distributed training, and a modular interface design, the framework substantially enhances computational efficiency and scalability. It enables effective attribution analysis for both extremely large language models and their pretraining datasets, significantly lowering the engineering barrier and thereby advancing both research and practical applications in the field.

Abstract

Data attribution is a promising field in interpretability that aims to explain model behavior through the influence of its training data, with applications including debugging undesirable model behavior and training dataset curation. However, significant engineering effort is required to perform it at scale, and many cutting edge techniques lack open-source tooling and support. Bergson is an open source library that aims to enable faster progress in the field by providing a host of techniques that scale to very large language models and pre-training datasets. The library natively supports on-disk gradient stores and multi-node distributed training, and provides quality of life tools for researchers. Finally, we introduce the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar. The library is available at https://github.com/EleutherAI/bergson .

Problem

• data attribution
• interpretability
• open-source library
• large language models
• scalability

Innovation

• data attribution
• open-source library
• large language models
• distributed training
• gradient storage