From Threads to Trajectories: A Multi-LLM Pipeline for Community Knowledge Extraction from GitHub Issue Discussions

2026-04-28

AI Summary

This work addresses the challenge developers face in efficiently extracting critical knowledge from fragmented and unstructured GitHub issue discussions within complex open-source projects. To tackle this, the authors propose an automated pipeline that leverages collaborative multi-agent large language models (LLMs) with a label-aware trajectory generation mechanism to transform raw discussions into structured problem-solving trajectories. By integrating external resource comprehension and role-specialized LLM coordination, the approach achieves, for the first time, high-fidelity reconstruction of expert-level reasoning paths from scattered dialogues. Evaluated on 800 real-world issues, the method successfully generates 734 high-quality trajectories (91.7% success rate), substantially enhancing the reusability of developer expertise and providing valuable, high-quality data for training LLM-based agents.

Abstract

Resolution of complex post-production issues in large-scale open-source software (OSS) projects requires significant cognitive effort, as developers need to go through long, unstructured and fragmented issue discussion threads before that. In this paper, we present SWE-MIMIC-Bench, an issue trajectory dataset generated from raw GitHub discussions using an automated multi-LLM pipeline. Unlike simple summarization, this pipeline utilizes a group of closed-source LLMs to perform granular tasks: analyzing individual comments with awareness of externally-linked resources, classifying comment analyses into label-specific fields (e.g., root cause, solution plan, implementation progress), and synthesizing label-aware trajectories which capture a structured and coherent narrative of the entire discussion thread. Our pipeline uses five closed-source LLM configurations for distinct purposes: label classification, inline code block and external link summarization, comment analysis, label-specific field classification and trajectory synthesis. By generating concise and reliable trajectories from complex conversation threads, this system can assist developers and researchers of broader software engineering community to understand the experience-driven collaborative approach for issue diagnosis. Furthermore, the generated trajectories can be used to train modern LLM agents to think and act like an expert developer. We evaluated our system on 800 real-world GitHub issues drawn from the SWE-Bench-Pro, SWE-Bench-Multilingual and SWE-Bench-Verified dataset, achieving a 91.7% success rate in extracting 734 high-fidelity reasoning trajectories.

Problem

• GitHub issue discussions
• knowledge extraction
• issue resolution
• software engineering
• discussion threads

Innovation

• Multi-LLM pipeline
• issue trajectory extraction
• structured knowledge synthesis
• label-aware narrative
• GitHub discussion analysis