LLM-based Content Classification Approach for GitHub Repositories by the README Files

2025-07-29

AI Summary

To address the low discoverability and usability of GitHub repositories caused by disorganized and incomplete README content, this paper proposes a fine-grained automatic classification method for README sections based on large language models (LLMs). Balancing performance and efficiency, we employ encoder architectures—including BERT, DistilBERT, and RoBERTa—integrated with the parameter-efficient fine-tuning technique LoRA, trained on a custom annotated dataset of 4,226 samples. Our framework achieves an overall F1-score of 0.98 on multi-class README section identification, significantly outperforming existing approaches. This work presents the first systematic validation of lightweight LLMs adapted via LoRA for structured parsing of open-source documentation. It demonstrates both technical feasibility and practical efficacy in enhancing repository searchability and supporting broader adoption of open-source projects.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

GitHub is the world's most popular platform for storing, sharing, and managing code. Every GitHub repository has a README file associated with it. The README files should contain project-related information as per the recommendations of GitHub to support the usage and improvement of repositories. However, GitHub repository owners sometimes neglected these recommendations. This prevents a GitHub repository from reaching its full potential. This research posits that the comprehensiveness of a GitHub repository's README file significantly influences its adoption and utilization, with a lack of detail potentially hindering its full potential for widespread engagement and impact within the research community. Large Language Models (LLMs) have shown great performance in many text-based tasks including text classification, text generation, text summarization and text translation. In this study, an approach is developed to fine-tune LLMs for automatically classifying different sections of GitHub README files. Three encoder-only LLMs are utilized, including BERT, DistilBERT and RoBERTa. These pre-trained models are then fine-tuned based on a gold-standard dataset consisting of 4226 README file sections. This approach outperforms current state-of-the-art methods and has achieved an overall F1 score of 0.98. Moreover, we have also investigated the use of Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) and shown an economical alternative to full fine-tuning without compromising much performance. The results demonstrate the potential of using LLMs in designing an automatic classifier for categorizing the content of GitHub README files. Consequently, this study contributes to the development of automated tools for GitHub repositories to improve their identifications and potential usages.

Problem

• Classify GitHub README file sections automatically
• Improve GitHub repository adoption via README quality
• Fine-tune LLMs for efficient text classification

Innovation

• Fine-tuned LLMs for GitHub README classification
• Used BERT, DistilBERT, RoBERTa models
• Applied LoRA for efficient fine-tuning