RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

2026-08-05

AI Summary

Current code comprehension benchmarks predominantly rely on bug reports from GitHub Issues, which are susceptible to being circumvented by language models through superficial pattern matching rather than genuine architectural understanding, and suffer from high subjectivity and severe editorial bias. To address these limitations, this work introduces RepoProbe—the first open-ended, repository-level architectural question-answering benchmark based on GitHub Discussions—alongside a checklist-based verification protocol that decomposes model responses into verifiable atomic facts for objective evaluation. By framing repository comprehension as an open-ended QA task, this approach reveals that while state-of-the-art large language models produce fluent answers, they often lack technical accuracy. The methodology quantitatively confirms the prevalence of editorial bias and significantly enhances the reliability of architectural understanding assessments.

Abstract

The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.

Problem

• repository comprehension
• edit bias
• LLM evaluation
• architecture understanding
• benchmarking

Innovation

• repository comprehension
• edit bias
• checklist-based verification
• LLM evaluation
• architecture-aware reasoning