🏟️ Long Code Arena (Project-level code completion)

This is the benchmark for Project-level code completion task as part of the 🏟️ Long Code Arena benchmark. Each datapoint contains the file for completion, a list of lines to complete with their categories (see the categorization below), and a repository snapshot that can be used to build the context. All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). The datapoints can be removed upon request.

How-to

Load the data via load_dataset:

from datasets import load_dataset

config_names = [
  'small_context',
  'medium_context',
  'large_context',
  'huge_context'
]

ds = load_dataset('JetBrains-Research/lca-project-level-code-completion', config_name, split='test')

Dataset Structure

Datapoints in the dataset have the following structure:

• repo – repository name in the format {GitHub_user_name}__{repository_name}
• commit_hash – commit hash of the repository
• completion_file – dictionary with the completion file content in the following format:
  • filename – path to the completion file
  • content – content of the completion file
• completion_lines – dictionary where the keys are categories of lines and values are a list of integers (numbers of lines to complete). The categories are:
  • committed – line contains at least one function or class from the files that were added on the completion file commit
  • inproject – line contains at least one function or class from the repository snapshot at the moment of completion
  • infile – line contains at least one function or class from the completion file
  • common – line contains at least one function or class with common names, e.g., main, get, etc.
  • non_informative – line that was classified to be non-informative, e.g., too short, contains comments, etc.
  • random – other lines.
• repo_snapshot – dictionary with a snapshot of the repository before the commit. It has the same structure as completion_file, but filenames and contents are orginized as lists.
• completion_lines_raw – same as completion_lines, but before sampling

How we collected the data

To collect the data, we cloned repositories from GitHub where the main language is Python. The completion file for each datapoint is a .py file that was added to the repository in a commit. The state of the repository before this commit is the repo snapshot.

The dataset configurations are based on the number of characters in .py files from the repository snapshot:

• small_context – less than 48K characters;
• medium_context – from 48K to 192K characters;
• large_context – from 192K to 768K characters;
• huge_context – more than 768K characters.

Datasets Stats

Completion File

Repository Snapshot .py files

Repository Snapshot non .py files

Line Counts:

Scores

You can find the results of running various models on this dataset in our leaderboard.

Citing

@article{bogomolov2024long,
  title={Long Code Arena: a Set of Benchmarks for Long-Context Code Models},
  author={Bogomolov, Egor and Eliseeva, Aleksandra and Galimzyanov, Timur and Glukhov, Evgeniy and Shapkin, Anton and Tigina, Maria and Golubev, Yaroslav and Kovrigin, Alexander and van Deursen, Arie and Izadi, Maliheh and Bryksin, Timofey},
  journal={arXiv preprint arXiv:2406.11612},
  year={2024}
}

You can find the paper here.