README
See also:
What is ENAMEL?
ENAMEL is a rigorous and high-standard benchmark for evaluating the capability of large language models (LLMs) in generating efficient code. We provide:
- A new metric eff@k characterizing the relationship between code efficiency and sample size k;
- A problem set consisting of 142 high-quality problems selected from OpenAI HumanEval;
- Expert-written efficient reference solutions, setting a high-standard for efficiency evaluation;
- Expert-written strong test case generators, enabling a rigorous evaluation of both correctness and efficiency;
- A Python library
enamfor easily evaluating the efficiency of LLM-generated code.
If you are interested in our work, please feel free to check our paper for detail.
Getting Started
For instructions on using this dataset, please check our GitHub repo.
LLM Leaderboard
The following table is a leaderboard of 30 LLMs (under greedy decoding) as well as HumanEval/HumanEval+ canonical solutions. Results show that LLMs fall short of generating expert-level efficient code. For more results, please refer to our paper.
We welcome LLM developers to submit their results to enrich this leaderboard. If you would like to submit your results, please organize your generated code samples into a .json file as described above and contact Ruizhong Qiu (rq5 AT illinois DOT edu).
| No. | Name | eff@1 | pass@1 |
|---|---|---|---|
| 1 | HumanEval+ | 0.517 | 0.958 |
| 2 | GPT-4 Turbo (Nov 2023) | 0.470 | 0.796 |
| 3 | HumanEval | 0.458 | 0.908 |
| 4 | GPT-4 (Jun 2023) | 0.454 | 0.831 |
| 5 | Llama 3 70B Instruct | 0.421 | 0.746 |
| 6 | Mixtral 8x22B Instruct | 0.408 | 0.746 |
| 7 | Claude 3 Opus | 0.401 | 0.789 |
| 8 | Phind Code Llama V2 | 0.394 | 0.683 |
| 9 | Claude 3 Haiku | 0.386 | 0.739 |
| 10 | ChatGPT | 0.364 | 0.683 |
| 11 | Claude 3 Sonnet | 0.345 | 0.662 |
| 12 | Llama 3 8B Instruct | 0.344 | 0.592 |
| 13 | Code Llama 34B Python | 0.268 | 0.458 |
| 14 | Mixtral 8x7B Instruct | 0.266 | 0.444 |
| 15 | Code Llama 70B Python | 0.264 | 0.500 |
| 16 | Code Llama 7B Python | 0.247 | 0.373 |
| 17 | Code Llama 13B Python | 0.216 | 0.408 |
| 18 | StarCoder | 0.195 | 0.352 |
| 19 | CodeGen 6B | 0.193 | 0.296 |
| 20 | CodeGen 16B | 0.169 | 0.310 |
| 21 | CodeT5+ 16B | 0.160 | 0.317 |
| 22 | CodeGen 2B | 0.153 | 0.254 |
| 23 | Mistral 7B | 0.152 | 0.275 |
| 24 | Vicuna 13B | 0.123 | 0.176 |
| 25 | SantaCoder | 0.100 | 0.141 |
| 26 | Incoder 6B | 0.091 | 0.127 |
| 27 | GPT-J | 0.083 | 0.106 |
| 28 | Incoder 1B | 0.066 | 0.092 |
| 29 | Vicuna 7B | 0.061 | 0.099 |
| 30 | GPT-Neo 2B | 0.043 | 0.056 |
| 31 | PolyCoder | 0.037 | 0.049 |
| 32 | StableLM 7B | 0.020 | 0.021 |
Acknowledgements
Contributors
19 commits
15 commits
Similar repos
Linked in READMEs
README
See also:
What is ENAMEL?
ENAMEL is a rigorous and high-standard benchmark for evaluating the capability of large language models (LLMs) in generating efficient code. We provide:
- A new metric eff@k characterizing the relationship between code efficiency and sample size k;
- A problem set consisting of 142 high-quality problems selected from OpenAI HumanEval;
- Expert-written efficient reference solutions, setting a high-standard for efficiency evaluation;
- Expert-written strong test case generators, enabling a rigorous evaluation of both correctness and efficiency;
- A Python library
enamfor easily evaluating the efficiency of LLM-generated code.
If you are interested in our work, please feel free to check our paper for detail.
Getting Started
For instructions on using this dataset, please check our GitHub repo.
LLM Leaderboard
The following table is a leaderboard of 30 LLMs (under greedy decoding) as well as HumanEval/HumanEval+ canonical solutions. Results show that LLMs fall short of generating expert-level efficient code. For more results, please refer to our paper.
We welcome LLM developers to submit their results to enrich this leaderboard. If you would like to submit your results, please organize your generated code samples into a .json file as described above and contact Ruizhong Qiu (rq5 AT illinois DOT edu).
| No. | Name | eff@1 | pass@1 |
|---|---|---|---|
| 1 | HumanEval+ | 0.517 | 0.958 |
| 2 | GPT-4 Turbo (Nov 2023) | 0.470 | 0.796 |
| 3 | HumanEval | 0.458 | 0.908 |
| 4 | GPT-4 (Jun 2023) | 0.454 | 0.831 |
| 5 | Llama 3 70B Instruct | 0.421 | 0.746 |
| 6 | Mixtral 8x22B Instruct | 0.408 | 0.746 |
| 7 | Claude 3 Opus | 0.401 | 0.789 |
| 8 | Phind Code Llama V2 | 0.394 | 0.683 |
| 9 | Claude 3 Haiku | 0.386 | 0.739 |
| 10 | ChatGPT | 0.364 | 0.683 |
| 11 | Claude 3 Sonnet | 0.345 | 0.662 |
| 12 | Llama 3 8B Instruct | 0.344 | 0.592 |
| 13 | Code Llama 34B Python | 0.268 | 0.458 |
| 14 | Mixtral 8x7B Instruct | 0.266 | 0.444 |
| 15 | Code Llama 70B Python | 0.264 | 0.500 |
| 16 | Code Llama 7B Python | 0.247 | 0.373 |
| 17 | Code Llama 13B Python | 0.216 | 0.408 |
| 18 | StarCoder | 0.195 | 0.352 |
| 19 | CodeGen 6B | 0.193 | 0.296 |
| 20 | CodeGen 16B | 0.169 | 0.310 |
| 21 | CodeT5+ 16B | 0.160 | 0.317 |
| 22 | CodeGen 2B | 0.153 | 0.254 |
| 23 | Mistral 7B | 0.152 | 0.275 |
| 24 | Vicuna 13B | 0.123 | 0.176 |
| 25 | SantaCoder | 0.100 | 0.141 |
| 26 | Incoder 6B | 0.091 | 0.127 |
| 27 | GPT-J | 0.083 | 0.106 |
| 28 | Incoder 1B | 0.066 | 0.092 |
| 29 | Vicuna 7B | 0.061 | 0.099 |
| 30 | GPT-Neo 2B | 0.043 | 0.056 |
| 31 | PolyCoder | 0.037 | 0.049 |
| 32 | StableLM 7B | 0.020 | 0.021 |
Acknowledgements
Similar repos
Linked in READMEs
Contributors
19 commits
15 commits