NEWS:

• A new version of the dataset with 120,000,000 more tokens is upload: OAK v1.1

Open Artificial Knowledge (OAK) Dataset

Overview

The Open Artificial Knowledge (OAK) dataset is a large-scale resource of over 650 Millions tokens designed to address the challenges of acquiring high-quality, diverse, and ethically sourced training data for Large Language Models (LLMs). OAK leverages an ensemble of state-of-the-art LLMs to generate high-quality text across diverse domains, guided by Wikipedia's main categories.

Key Features

• 653,552,076 tokens of high quality synthetic data
• Generated using GPT4o, LLaMa3-70B, LLaMa3-8B, Mixtral-8x7B, Gemma-7B, and Gemma-2-9B
• Broad knowledge coverage
• Designed to foster the development of more capable and aligned language models

Download

from datasets import load_dataset

ds = load_dataset("tabularisai/oak", split="train")
ds[0]

Dataset Creation Process

1. Subject Extraction: High-level topics are extracted from Wikipedia.
2. Subtopic Expansion: Topics are expanded into detailed subtopics using advanced language models like GPT-4o.
3. Prompt Generation: Prompts are created using programming prompt engineering and meta-prompt techniques.
4. Text Generation: Content is generated using various open-source LLMs.

Future Work

• Increase dataset volume
• Add more languages
• Incorporate more advanced and diverse models
• Refine the dataset's application in code-related tasks
• Foster community contributions

Citation

@misc{borisov2024open,
      title={Open Artificial Knowledge}, 
      author={Vadim Borisov and Richard H. Schreiber},
      year={2024},
      eprint={2407.14371},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.14371}, 
}

Links

• Webiste: www.oakdataset.org
• Paper: https://arxiv.org/abs/2407.14371
• GitHub: https://github.com/tabularis-ai/oak-dataset

Disclaimer

Users must adhere to ethical guidelines, respect privacy considerations, and be mindful of potential biases in the synthetic data. The OAK dataset is intended for research purposes only.

Contact

For questions or more data, please contact: info@tabularis.ai

www.tabularis.ai