EverythingLM V2 Dataset

EverythingLM V2 is a diverse instruct dataset consisting of 1k of human-assistant conversations. These sets were generated using principles from both evol-instruct and Orca. The dataset encompasses a wide array of topics and interactions.

Differences for V1:

• All data in V2 is generated by GPT4
• Higher quality dataset generation pipeline:
  • More humalike seed prompts
  • Fixed some bugs in the script
  • More diverse creative writing
  • More diverse seed prompts in general
  • Attempt not to overfit the model on complex instructions by occasionally skipping evol

Cost:

Reproducing this dataset would cost roughly $40.

Instruction Categories:

• Reasoning
• Creative Writing
• General Knowledge
• Brainstorming
• Search Query
• Coding
• Basic Instruct

We also leverage various system prompts for evol-instruct and for responding to prompts. This dataset has also been filtered to remove OpenAI alignment.

How it stands out:

• Long, detailed outputs
• Humanlike creativity
• CoT reasoning
• Complex & challenging tasks

Plans:

• Train Llama 7b & 13b models (13b model V1 trained)
• Train Llama 70b QLoRA
• Generate V2 of the dataset, with more categories and GPT-4 (DONE) ✓

Included in this repo is the script to generate the dataset.