ECCO

Dataset from the paper "ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?"

The dataset consists of 2 subsets edit and generate each with 3 splits (train, val and test).

Code repository: https://github.com/CodeEff/ECCO

Loading the dataset / benchmark

dataset = load_dataset('CodeEff/ECCO', 'edit') # For history-based editing setting
dataset = load_dataset('CodeEff/ECCO', 'generate') # For nl-instructed generation setting

These are used to generate code by each model across the 2 paradigms. We use the test split for the evaluation/results and the train and val splits for finetuning and few-shot prompting.

Download the test cases

mkdir data && cd data
wget https://huggingface.co/datasets/CodeEff/ECCO/resolve/main/test_cases.zip
unzip test_cases.zip

Evaluation dataset

The dataset also consists of an additional 3rd subset generate_eval which consists of the runtime and memory of a spectrum of user solutions for each problem in the test split.
This is used for the percentile evaluation of the NL-instructed generation paradigm.

Data Sources

Dataset is sourced from IBM CodeNet which consists of primarily competetive programming solutions. This is further filtered for efficiency and correctness as described in our paper.

Citation

@article{waghjale2024ecco,
  title={ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?},
  author={Waghjale, Siddhant and Veerendranath, Vishruth and Wang, Zora Zhiruo and Fried, Daniel},
  journal={arXiv preprint arXiv:2407.14044},
  year={2024}
}