ORCA-Chat

A high-quality explanation-style chat dataset.

ORCA dataset is cool, but it cannot directly be used to finetune chat models with above 4k context length because it has trivial samples with tokens above 4k. It also has a large number of redundant instructions which degrades its quality and increases the compute time when finetuning models using it. Enter ORCA-Chat!

This is a cleaned, pruned, and clustered version of orca to form a conversation-style dataset. The the process involves removing samples with very high similarity and also grouping instructions to form conversation.

What next?

I will release 16/32k versions for this soon!

Credits

• This wouldn't be possible without the amazing work of Eric in recreating the ORCA dataset. Check it out: https://huggingface.co/datasets/ehartford/dolphin
• This dataset was created in association with the Open-Assistant team @jordanclive and @andreaskoepf

Citations

@misc{Orca-Chat,
  title = {Orca-chat: A high-quality explanation-style chat dataset.},
  author = {Shahul Es},
  year = {2023},
  publisher = {HuggingFace},
  journal = {HuggingFace repository},
  howpublished = {\url{https://huggingface.co/datasets/shahules786/orca-chat/},
}