Contributors: Nicolas Mejia Petit

Vezora's CodeTester Dataset

Introduction

Today, on March 6, 2024, we are excited to release our internal Python dataset with 143,327 examples of code. These examples have been meticulously tested and verified as working. Our dataset was created using a script we developed.

Dataset Creation

• Our script operates by extracting Python code from the output section of Alpaca-formatted datasets. It tests each extracted piece of code, keeping it if it passes and removing it if it fails, then saves all the working code in a seperate dataset.
• Our second script works by removing the not working code from your alpaca datasets, and saves it to a not working code json, and then keeps all the working examples along with any other non python related examples, and saves it.
• !WARNING! these scripts run on your local computer's python enviroment, with mutithreading so it runs fast, if there is any malicious python code in your dataset, it WILL run on your local computer so run it in a VM. Lastly, it is required that you have python packages installed, just main ones most would have already installed but some like tkinter and other packages in order for certain lines of code to be tested.
• (if you are struggling converting your dataset to alpaca format, give the first three questions of both datasets and ask chat gpt or bing to give you a script to convert the dataset to that format you want. Might take one or two tries.)
• The creation of this dataset involved leveraging open source datasets from various sources, including Wizard-LM's Evol datasets, CodeUp's 19k, Sahils2801's Code Alpaca, Eric Heartford's Dolphin and dolphin coder sharegpt 290k, both ise uiuc's Magicoder dataset, and a selection of hand-prompted GPT-4 code questions. The resulting dataset was carefully deduplicated.
• We discovered that many of the open source datasets contained thousands of non-functional code examples, often plagued by module errors and other issues. Importantly, our script's approach is highly adaptable and could potentially be used to test code in other languages such as C++, C, SQL, and more.

Usage Guidelines

We invested a significant amount of time in developing this script. If you intend to use it to extract functional code in your own projects or datasets, and or plan on using our dataset, please include the following attribution in your model's or dataset's repository:

"Filtered Using Vezora's CodeTester"

Motivation

many openly trained datasets are contaminated with code that does not run. Code that you will be teaching a model, degrading its preformance. We release this dataset to suit as a large corpus of working python instruction and answer dataset. This current itteration has 141k examples of working code. Allowing a model to learn a wide range of python tasks as best as possible.

Limitations of Foundational Models

It's essential to note that even when writing syntactically correct code, foundational models often lack access to up-to-date Python and API documentation. As a result, code generated by these models may contain errors stemming from outdated calls or methods.

Building a Strong Python Code Model

If you aspire to build a robust Python code model, we recommend the following steps:

1. Pretrain with Mistral 7b on UPTODATE Python and API documentations. (during our testing we found even when a model writes syntactyically correct code it lacks up to date api calls and functions.)
2. Consider incorporating programming textbooks into your training.
3. Fine-tune your model with our dataset using SFT (Supervised Fine-Tuning).

In the future, we may also release our "not working" code dataset, allowing users to create a Discriminative Pretraining Objective (DPO) model to reward functional code over non-functional code. Although with the second script provided, it would be pretty easy to do it your self.

We hope this dataset serves as a valuable resource for the community and contributes to the improvement of code-related AI models.