OSS License Identification at Scale: A Comprehensive Dataset Using World of Code

2024-09-07

arXiv.org

AI Summary

License identification in open-source software supply chains faces challenges of scale, heterogeneous reuse, and dynamic evolution. To address this, we introduce the first large-scale, temporally annotated, fine-grained license identification dataset. Leveraging the World of Code infrastructure, we scan files containing “license” in their paths; then apply the Winnowing algorithm combined with SPDX standards for approximate matching, identifying 5.5 million distinct license snippets. We further construct a project-to-license (P2L) temporal mapping covering the entire GitHub commit history. Our proposed scalable identification paradigm integrates path-based heuristics with text-based approximate matching. Evaluated via stratified sampling and manual validation, it achieves 92.08% accuracy and an F1-score of 91.11%. The dataset is publicly released to support compliance auditing, license evolution analysis, and tool development.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The proliferation of open source software (OSS) and different types of reuse has made it incredibly difficult to perform an essential legal and compliance task of accurate license identification within the software supply chain. This study presents a reusable and comprehensive dataset of OSS licenses, created using the World of Code (WoC) infrastructure. By scanning all files containing"license"in their file paths, and applying the approximate matching via winnowing algorithm to identify the most similar license from the SPDX list, we found and identified 5.5 million distinct license blobs in OSS projects. The dataset includes a detailed project-to-license (P2L) map with commit timestamps, enabling dynamic analysis of license adoption and changes over time. To verify the accuracy of the dataset we use stratified sampling and manual review, achieving a final accuracy of 92.08%, with precision of 87.14%, recall of 95.45%, and an F1 score of 91.11%. This dataset is intended to support a range of research and practical tasks, including the detection of license noncompliance, the investigations of license changes, study of licensing trends, and the development of compliance tools. The dataset is open, providing a valuable resource for developers, researchers, and legal professionals in the OSS community.

Problem

• Accurate identification of OSS licenses in software supply chains.
• Creation of a comprehensive dataset using World of Code infrastructure.
• Support for research on license compliance, changes, and trends.

Innovation

• Uses World of Code for OSS license dataset creation
• Applies winnowing algorithm for license identification
• Provides dynamic P2L map with commit timestamps