An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation

2025-05-26

AI Summary

This paper addresses the high inference cost of large language models (LLMs) in warehouse-scale code generation. We propose a collaborative framework integrating strong and weak LLMs. Methodologically, we design a context-distribution-based, pipeline-orchestrated, and dynamically routed collaboration architecture, coupled with an empirical evaluation framework to systematically analyze cost–performance trade-offs across diverse collaboration paradigms on GitHub issue resolution. Our key contributions are: (1) identification and validation of a dynamic collaboration strategy that matches the performance of the strongest standalone model while reducing inference cost by 40%; (2) the first empirical demonstration that weak models exhibit substantial capability gains under structured collaboration, with context distribution and pipeline orchestration emerging as the most effective enhancement mechanisms; and (3) practical, deployment-oriented guidelines for model selection and collaboration strategy design, accompanied by open-sourcing of the complete codebase.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study cost-efficient collaboration between strong and weak language models for repository-level code generation, where the weak model handles simpler tasks at lower cost, and the most challenging tasks are delegated to the strong model. While many works propose architectures for this task, few analyze performance relative to cost. We evaluate a broad spectrum of collaboration strategies: context-based, pipeline-based, and dynamic, on GitHub issue resolution. Our most effective collaborative strategy achieves equivalent performance to the strong model while reducing the cost by 40%. Based on our findings, we offer actionable guidelines for choosing collaboration strategies under varying budget and performance constraints. Our results show that strong-weak collaboration substantially boosts the weak model's performance at a fraction of the cost, pipeline and context-based methods being most efficient. We release the code for our work at https://github.com/shubhamrgandhi/codegen-strong-weak-collab.

Problem

• Cost-efficient collaboration between strong and weak language models for code generation
• Evaluating collaboration strategies for GitHub issue resolution performance vs cost
• Reducing costs by 40% while maintaining strong model performance

Innovation

• Strong-weak model collaboration for cost efficiency
• Dynamic delegation of tasks based on complexity
• Pipeline and context-based methods optimize performance