Dataset Card for FEA-Bench

A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation.

Dataset Details

Dataset Description

The FEA-Bench is a benchmark with a test set that contains 1,401 task instances from 83 Github repositories. This benchmark aims to evaluate the capabilities of repository-level incremental code development. The task instances are collected from Github pull requests, which have the purpose of new feature implementation. Each task instance includes the repo and the base commit sha256, and the PR number and the status of unit test.

• Curated by: the authors of the FEA-Bench paper: Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao and their collaborators.
• Language(s) (NLP): English
• License: Others; We list all licenses of involved github repositories in the last part.

Uses

This dataset is designed to evaluate performances of LLMs on repository-level code development, which is a complicated software engineering task.

• Repository-level incremental code development: The FEA-Bench can be used to evaluate a model for the the capabilities of repository-level incremental code development. Success on this task is typically measured by achieving a high/low resolved ratio. The leaderboard will soon be published as a website.

Direct Use

Use scripts from FEA-Bench repo to get info for task instances and organize them into prompt, which can be used to LLMs' inference. Also, you can get info or use agents to directly solve the PRs with code changes.

Out-of-Scope Use

This dataset is not aimed at training for LLMs. You should not take the FEA-Bench as the training dataset to avoid contamination.

Dataset Structure

An example:

{
    "instance_id": "huggingface__accelerate-270",
    "pull_number": 270,
    "repo": "huggingface/accelerate",
    "version": null,
    "base_commit": "515fcca9ed2b36c274c595dbdff75f1c2da635de",
    "environment_setup_commit": "08101b9dde2b1a9658c2e363e3e9f5663ba06073",
    "FAIL_TO_PASS": [
        "tests/test_state_checkpointing.py::CheckpointTest::test_can_resume_training",
        "tests/test_state_checkpointing.py::CheckpointTest::test_invalid_registration",
        "tests/test_state_checkpointing.py::CheckpointTest::test_with_scheduler"
    ],
    "PASS_TO_PASS": []
}

Dataset Creation

Curation Rationale

Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories.

Source Data

Data Collection and Processing

We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified.

Who are the source data producers?

Authors of 83 Github repositories list in the last part.

Personal and Sensitive Information

The dataset does not include any personal or sensitive information.

Bias, Risks, and Limitations

• The quantity of high-quality data suitable for repository-level incremental development is limited. High-quality and usable pull requests for new feature development are relatively scarce. Many repository-level code developments for implementing new functionalities were committed during the early stages of repositories, without going through the rigorous code review process typical of the open-source community, resulting in lower data quality that cannot be utilized.
• Furthermore, the software's early-stage developments might not even have been conducted using the GitHub platform, posing a challenge for data collection and utilization.
• The repository-level incremental code development may not just include new feature implementation tasks.
• Only Python repositories are involved in FEA-Bench.
• The inference results of the task instances from the benchmark may contain code that is harmful to computer systems.

Recommendations

Evaluation by docker is recommended, just like SWE-bench. We will also publish a patch for SWE-bench to make it compatible for our tasks' evaluation.

BibTeX:

To be appeared after publishing the ArXiv paper.

APA:

To be appeared after publishing the ArXiv paper.

Dataset Card Contact

For further information or questions, please contact Xin Zhang (xinzhang3@microsoft.com).

All involved Github repositories in the FEA-Bench