Contributors: Nicolas Mejia Petit

Vezora's CodeTester Dataset

Introduction

Today, on November 2, 2023, we are excited to release our internal Python dataset with 22,600 examples of code. These examples have been meticulously tested and verified as working. Our dataset was created using a script we developed.

Dataset Creation

• Our script operates by extracting Python code from the output section of Alpaca-formatted datasets. It tests each extracted piece of code, keeping it if it passes and removing it if it fails, then saves all the working code in a seperate dataset.
• Our second script works by removing the not working code from your alpaca datasets, and saves it to a not working code json, and then keeps all the working examples along with any other non python related examples, and saves it.
• !WARNING! this script does run on ypur local computer, with mutithreading so it runs fast, if there is any malicious python code in your dataset, it WILL run on your local computer so either run in a VM or don't sift through shady datasets. Lastly, it is required that you have python packages installed, just main ones most would have already installed but some like tkinter and other packages in order for certain lines of code to be tested.
• (if you are struggling converting your dataset to alpaca format, give the first three questions of both datasets and ask chat gpt or bing to give you a script to convert the dataset to that format you want. Might take one or two tries.)
• The creation of this dataset involved leveraging open source datasets from various sources, including Wizard-LM's Evol datasets, CodeUp's 19k, Sahils2801's Code Alpaca, Eric Heartford's Dolphin, and a selection of hand-prompted GPT-4 code questions. The resulting dataset was carefully deduplicated.
• We discovered that many of the open source datasets contained thousands of non-functional code examples, often plagued by module errors and other issues. Importantly, our script's approach is highly adaptable and could potentially be used to test code in other languages such as C++, C, SQL, and more.

Usage Guidelines

We invested a significant amount of time in developing this script. If you intend to use it to extract functional code in your own projects or datasets, and or plan on using our dataset, please include the following attribution in your model's or dataset's repository:

"Filtered Using Vezora's CodeTester"

Motivation

We are releasing our internal tool thanks to Open Chat 3.5's recognition of its foundational model limitations, particularly in tasks related to code.

Limitations of Foundational Models

It's essential to note that even when writing syntactically correct code, foundational models often lack access to up-to-date Python and API documentation. As a result, code generated by these models may contain errors stemming from outdated calls or methods.

Building a Strong Python Code Model

If you aspire to build a robust Python code model, we recommend the following steps:

1. Pretrain with Mistral 7b on UPTODATE Python and API documentations. (during our testing we found even when a model writes syntactyically correct code it lacks up to date api calls and functions.)
2. Consider incorporating programming textbooks into your training.
3. Fine-tune your model with our dataset using SFT (Supervised Fine-Tuning).

In the future, we may also release our "not working" code dataset, allowing users to create a Discriminative Pretraining Objective (DPO) model to reward functional code over non-functional code. Although with the second script provided, it would be pretty easy to do it your self.

We hope this dataset serves as a valuable resource for the community and contributes to the improvement of code-related AI models.

Why there are some references to 188k, we had used a script to count the examples in the dataset, and not realized the script wasn't meant to alpaca datasets, so it counted the examples wrong. Therefore, this is "only" 22k of functioning python code examples. However we are soon to release a better coding dataset, people will be even very happy with, containing over 220,000 examples of code (only tested for python code but contains many other languages.) I will also be releasing 13k examples of not working code, for the purpose of a DPO datasets, or RLHF.