LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

LongCodeBench (LCB) is a benchmark designed to test Large Language Model (LLM) coding abilities in long-context scenarios. As presented in the paper "LongCodeBench: Evaluating Coding LLMs at 1M Context Windows", it addresses the challenge of constructing realistic long-context benchmarks by drawing from real-world GitHub issues to create both QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. The benchmark aims to evaluate models across different scales, revealing limitations and potential areas for improvement in long-context code comprehension and repair.

• Paper: LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
• Code: https://github.com/Zteefano/long-code-bench

Abstract

Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at this https URL and the codebase to replicate the work on this paper at this https URL .

Dataset Description

LongCodeBench provides a comprehensive evaluation framework for large language models in scenarios requiring extensive code context. It comprises two main sub-benchmarks:

• LongSWE-Bench: A tunable version of the SWE-Bench benchmark, where each problem statement is repeated with a varying number of context files. This allows for testing models' coding capabilities at different context lengths, specifically focusing on bug fixing and code repair tasks derived from real-world GitHub issues.
• LongCodeQA: A question-answering task designed to test code comprehension. It is built from real-world GitHub issues, challenging models to understand and respond to inquiries within long-context settings.

The benchmark's design enables evaluation across various model scales and helps identify strengths and weaknesses of LLMs in handling large codebases for both repair and comprehension.

Sample Usage

The LongCodeBench repository uses the Pixi package manager for environment setup. The following instructions, extracted from the GitHub repository, detail how to initialize the environment and run various tasks.

Initialization

First, install Pixi. If on Linux or macOS, run:

curl -fsSL https://pixi.sh/install.sh | bash

Once Pixi is installed (restarting the terminal may be needed), install all necessary dependencies:

pixi install

API Keys

Create an .env file to store required API keys, for example:

HF_TOKEN=[Hugging Face Token for gated models]
OPENAI_API_KEY=[OpenAI API Key]

Create Tunable SWE-Bench Dataset

To create a tunable version of a SWE-Bench dataset (e.g., from princeton-nlp/SWE-bench_Verified), refer to the src/swe_bench/swebench/inference/make_datasets/README.md for retrieval file generation, then run:

pixi r make_swebench_tuned \
	--dataset princeton-nlp/SWE-bench_Verified \
	--splits test \
	--output_dir [directory_store_dataset] \
	--prompt_style style-3 \
	--max_k 20 \
	--retrieval_type "bm25"

Running Evaluations

For running an evaluation on a dataset, use the eval task with Pixi:

pixi r eval \
	dataset=[dataset_file] \
	model=gpt4omini \
	output=[results_file_path]

Harnessing Evaluations

After patches for a set of instances have been generated (e.g., for LongSWE-Bench), it is possible to harness their performance with the harness_tuned task:

pixi r harness_tuned \
	--dataset [dataset_hf_identifier] \
	--predictions_path [results_file_path] \
	--max_workers [num_workers_to_use] \
	--run_id [unique_run_id] \
	--output_file [harness_results_path]

Build CodeQA Task

To build the CodeQA task from a list of repositories, create a file (e.g., repos_list.txt) with owner/repo_name on each line, then run:

pixi r make_qa \
	--repos [repositories_list_file] \
	--output [output_directory] \
	--format [prompt_format]

Example repositories_list_file content:

yaml/pyyaml
pypa/virtualenv
jaraco/zipp

Evaluating CodeQA

The same eval task used above can be used for LongCodeQA, ensuring the dataset files (e.g., conf/dataset/codeqa/32K.yaml) have the task_type property set to longcodeqa:

pixi r eval \
	dataset=[dataset_file] \
	model=gpt4omini \
	output=[results_file_path]

Citation

If you use LongCodeBench in your research, please cite the following paper:

@misc{rando2025longcodebenchevaluatingcodingllms,
      title={LongCodeBench: Evaluating Coding LLMs at 1M Context Windows},
      author={Stefano Rando and Luca Romani and Alessio Sampieri and Luca Franco and John Yang and Yuta Kyuragi and Fabio Galasso and Tatsunori Hashimoto},
      year={2025},
      eprint={2505.07897},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.07897},
}