Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks

2025-05-22

AI Summary

Existing open-source large language models (LLMs) perform well on function-level code generation but struggle with repository-level software engineering tasks requiring cross-file and cross-function semantic and structural dependency understanding. Moreover, mainstream agent-based approaches suffer from privacy risks, limited controllability, and poor accessibility. To address these challenges, we propose the first end-to-end graph-enhanced framework that explicitly injects code graph structure into the LLM’s attention mechanism. Our approach employs a lightweight adapter to map graph node attributes into the language model’s input space and synergistically integrates graph neural network embeddings with graph-augmented retrieval-augmented generation (RAG). Evaluated on SWE-bench Lite using Qwen2.5-72B, our method achieves a 43.00% task resolution rate—the highest among all open-source models to date—and outperforms the previous state-of-the-art by 12.33 percentage points, ranking eighth overall.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recent advances in Large Language Models (LLMs) have shown promise in function-level code generation, yet repository-level software engineering tasks remain challenging. Current solutions predominantly rely on proprietary LLM agents, which introduce unpredictability and limit accessibility, raising concerns about data privacy and model customization. This paper investigates whether open-source LLMs can effectively address repository-level tasks without requiring agent-based approaches. We demonstrate this is possible by enabling LLMs to comprehend functions and files within codebases through their semantic information and structural dependencies. To this end, we introduce Code Graph Models (CGMs), which integrate repository code graph structures into the LLM's attention mechanism and map node attributes to the LLM's input space using a specialized adapter. When combined with an agentless graph RAG framework, our approach achieves a 43.00% resolution rate on the SWE-bench Lite benchmark using the open-source Qwen2.5-72B model. This performance ranks first among open weight models, second among methods with open-source systems, and eighth overall, surpassing the previous best open-source model-based method by 12.33%.

Problem

• Addressing repository-level software engineering tasks with open-source LLMs
• Overcoming unpredictability and accessibility issues of proprietary LLM agents
• Enhancing LLM comprehension of codebases via semantic and structural dependencies

Innovation

• Integrates code graph structures into LLM attention
• Uses adapter to map node attributes
• Agentless graph RAG framework boosts performance