Dataset Card for H4 Code Evaluation Prompts

These are a filtered set of prompts for evaluating code instruction models. It will contain a variety of languages and task types. Currently, we used ChatGPT (GPT-3.5-tubro) to generate these, so we encourage using them only for qualatative evaluation and not to train your models.

The generation of this data is similar to something like CodeAlpaca, which you can download here, but we intend to make these tasks both
a) more challenging, and b) more curated.

These two things hopefully give a meaningful evaluation, but is not enough data to train an entire model.

The data corresponds to the following:

• 20 simple python instruction following,
• 20 intermediate python instruction following,
• 10 advanced python instruciton following,
• 15 python machine learning questions,
• 20 C++ instruction following,
• 10 html instruction following,
• 20 misc language code feedback questions.

Or, on a per language basis:

• Python: 81
• C++: 21
• html: 10
• Ruby: 1
• Bash: 1
• MATLAB: 1
• React: 1
• Scala: 1
• JavaScript: 1
• Java: 1
• PHP: 1

Or, per instruction type:

• Code completion / instruction following: 95
• Bug fixing: 20

To get the current information on the tasks, you can use the following snippet:

from datasets import load_dataset
d = load_dataset("HuggingFaceH4/code_evaluation_prompts")
language_list = d['train']['language']
language_count = {ele:language_list.count(ele) for ele in language_list}

Similar code can be run for the type of instruction (code generation vs. bug advice).

Interested in contributing? Open a PR with a specific language and question content.

Here are the ChatGPT prompts used to initiate the responses (which are then filtered), May 3rd 2023 version:

• Generate a bunch of instructions for coding questions in python (in the format of {"prompt": instruction})
• These have been useful, can you generate the last few that are the hardest and most Pythonic that you can think of?
• Taking a step back, can you generate 20 for me that don't need to be as hard, but are machine learning focused (e.g. a mix of PyTorch and Jax).
• Generate a bunch of instructions for coding questions in C++ (in the format of {"prompt": instruction})
• Can you generate 5 examples of instructions, with the same format {"prompt": text}, where the instruction has a piece of code with a bug, and you're asking for feedback on your code as if you wrote it?