🌱⚙️ EnvBench

This repository contains data associated with EnvBench benchmark from EnvBench: A Benchmark for Automated Environment Setup.

It contains:

• statistics about repositories from GitHub Search under ghs/data folder;
• several data splits under splits folder;
• Git repositories under repos folder;
• READMEs under readmes folder (available as readmes config);
• GitHub Actions workflows under workflows folder (available as workflows config);
• list of files in the repositories (available as repo_contents config).

Data splits

We provide the repositories at each step of our filtering process.

JVM

1. We obtain the list of repositories from GitHub Search: ghs/data/jvm_permissive.jsonl
2. We clone the repositories to repos/full/jvm
3. We filter repositories from previous step to keep only those with configuration files related to a single build tool in root directory: splits/jvm_single_dm_in_root_folder.jsonl (splits/jvm_single_dm_in_root_folder.jsonl additionally contains 'dm' field with build tool for each repository)
4. We filter the repositories from previous step to keep only those without Docker-related configuration files: splits/jvm_single_dm_in_root_folder_no_docker.jsonl
5. We filter the repositories from previous step to keep only those that our deterministic script couldn't configure correctly: splits/jvm_baseline_failure.jsonl

The list of 665 repositories from our benchmark is available in splits/jvm_baseline_failure.jsonl. For convenience, the final set of Git repositories is also stored separately under repos/final/jvm.

Python

1. We obtain the list of repositories from GitHub Search: ghs/data/python_permissive.jsonl
2. We clone the repositories to repos/full/python
3. We filter repositories from previous step to keep only those with configuration files related to a single build tool in root directory: splits/python_single_dm_in_root_folder.jsonl (splits/python_single_dm_in_root_folder.jsonl additionally contains 'dm' field with dependency manager for each repository)
4. We filter the repositories from previous step to keep only those without Docker-related configuration files: splits/python_single_dm_in_root_folder_no_docker.jsonl
5. We filter the repositories from previous step to keep only those that our deterministic script couldn't configure correctly: splits/python_baseline_failure.jsonl

The list of 329 repositories from our benchmark is available in splits/python_baseline_failure.jsonl. For convenience, the final set of Git repositories is also stored separately under repos/final/python.

📚 Citation

If you find this work useful for your research, please consider the following citation:

@inproceedings{eliseeva2025envbench,
    title={EnvBench: A Benchmark for Automated Environment Setup},
    author={Aleksandra Eliseeva and Alexander Kovrigin and Ilia Kholkin and Egor Bogomolov and Yaroslav Zharov},
    booktitle={ICLR 2025 Third Workshop on Deep Learning for Code},
    year={2025},
    url={https://openreview.net/forum?id=izy1oaAOeX}
}