Papers for Repo-Level Code Generation

This repo maintains the list of papers for repo-level code generation.

Feel free to create pull request to add more.

Benchmark

1. 12/2024: FullStack Bench: Evaluating LLMs as Full Stack Coders

   • Benchmark for full-stack programming in many domains, and also release sandbox tool to evaluate.

2. 10/2024: RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems

   • Data mostly extracted from GitHub, task designed as retrieval and completion, etc. Using matching accuracy as evaluation metric.

3. 10/2024: EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations

   • A dynamically evolving code generation benchmark designed to address data leakage and provide domain-specific evaluations by periodically updating datasets.

4. 6/2024: REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark

   • executable python benchmark focuses on the pass rate of repository function level code generation

5. 2023, SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

   • Well-known benchmark to evaluate the agent's capabilties.

6. 2023, RepoEval: Repocoder: Repository-level code completion through iterative retrieval and generation

   • Similar to RepoBench, data is also extracted from GitHub

Papers

1. 12/2025: Natural Language Summarization Enables Multi-Repository Bug Localization by LLMs in Microservice Architectures (ICSE 2026 Workshop)
   • Proposes reframing bug localization as a natural language reasoning task by transforming codebases into hierarchical NL summaries and performing NL-to-NL search.
2. 06/2025: Repository-Level Code Understanding by LLMs via Hierarchical Summarization: Improving Code Search and Bug Localization (ICCSA 2025)
   • Enabling LLMs to Understand Code at Repository-level using Hierarchical Summarization for Code Search and Bug Localization
3. 3/2025: What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
   • An emperical study on the retrieved information of current works and their effectiveness. Proposed AllianceCoder, which decomposes user query into query on invoked API. Then retrieve those APIs and use them to generate function-level code.
4. 2024: RLCoder: Reinforcement Learning for Repository-Level Code Completion (ICSE 2025)
   • Using RL to implement Repo-level code completion based on RAG method.
5. 2024: EVOR: Evolving Retrieval for Code Generation
   • Design a way to improve the quality of the knowledge for retrieval. Overall, the framework is RAG for code generation
6. 11/2024: Repository-level Code Translation Benchmark Targeting Rust
   • Work on repo-level code translation task? (not sure about this task, it seems translating into test cases)
7. 10/2024: Repository-Level Compositional Code Translation and Validation
   • Design a pipeline for code translation
8. 10/2024: M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
   • A benchmark and also an instruction-tuning dataset for code completion.
9. 06/2024: How to Understand Whole Software Repository
   • Try to resolve the task in SWE-Bench, using some ASE methods.
10. 06/2024: Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
    • iteratively retrieve the context based on the compiler feedback
11. 06/2024: GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model
    • Slicing the blocks/lines of code as retrieved context
12. 06/2024： R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models
    • A repo-level completion benchmark and a context retrieval and prompt assemble powered code completion framework.
13. 06/2024： Enhancing Repository-Level Code Generation with Integrated Contextual Information
    • Designed for statically typed programming languages. Integrate relevant code and type context.
14. 05/2024: Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion
    • ACL-2024 accepted: extract entities and relations formalism, to obtain the context graph
15. 03/2024: Repoformer: Selective Retrieval for Repository-Level Code Completion
    • A pre-training approach to tackle repo-level code retrieval
16. 02/2024: Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Context
    • Leverage the IDE cross-file information for LLM to perform repo-level code generation.
17. 01/2024: CODEAGENT: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (from software engieerning)
    • propose a benchmark for evaluation.
    • Method: use tool to retrieve, rather than similarity
18. 12/2023: Context-Aware Code Generation Framework for Code Repositories: Local, Global, and Third-Party Library Awareness (from software engieerning)
    • Focus on enhancing the retrieval process (based on GPT-3.5-Turbo)
19. 11/2023：ML-BENCH: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
20. 11/2023: Guiding Language Models of Code with Global Context using Monitors
    • Maintain a monitor while performing repo-level code generation
21. 10/2023: CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
    • benchmark that required cross-file reasoning
22. 09/2023: CodePlan: Repository-level Coding using LLMs and Planning
    • plan first, then execute
23. 06/2023: RepoFusion: Training Code Models to Understand Your Repository
    • trained to understand the whole repo
24. 03/2023: RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (EMNLP 2023)
    • Iteratively retrieve code from repo based on similary, until the code is correct
25. 03/2023: InferFix: End-to-End Program Repair with LLMs (ICSE)
    • query the database to retrieve
26. 06/2022: Repository-Level Prompt Generation for Large Language Models of Code (ICML 2023)
    • generate prompt based on the complete repo, classify from a list of prompt proposal

Survey Papers

1. 06/2024: A Survey on Large Language Models for Code Generation
2. 01/2024: If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

Relevant Agent-based Papers

1. 08/2023: MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
2. 07/2023: Communicative Agents for Software Development