KoRAE Dataset

We used filtered high-quality Korean dataset for finetuning of KoRAE. First of all, we gathered Korean data and make the mixture of them. Then we filtered high-quality data from the combination of data through filtering method that introduced from AlpaGasus. The overview of data processing procedure is as follws:

1. Collect various Korean dataset from HuggingFace Hub.
2. Rate the data quality using gpt-3.5-turbo.
3. Process the rated data and filter the high-scored data.

Let's go deeper into data processing!

1. Korean dataset mixture

We investigated several sources to collect high-quality Korean data, and among them, we collected data from the various sources. As a result, we were able to create a new dataset containing 64K pieces of data. The specific configuration of the dataset is as follows:

You can check the original KoRAE dataset here: KoRAE_original

2. Rating

We utilized ChatGPT(gpt-3.5-turbo) as rater to rate the quality of dataset. We considered whether to use the prompt for the evaluation in Korean or English, but we thought it would be undesirable to give evaluations in different languages, so we conducted the evaluation using the Korean prompt. The overall data rating method was reffered to AlpaGasus

3. Processing & Filtering

We postprocessed rated dataset after the rating. The main postprocessing procedure are as follows:

• Wrong score extraction correction
• Incorrect format dataset exclusion

You can check the postprocessed KoRAE dataset here: KoRAE_rated_filtered

After the all postprocessing, we analysed the score distribution of rated dataset. As shown in the following figure, it was confirmed that 8-point data was the most. This confirms that KoRAE dataset consisted of high-quality data from the beginning.

However, We filtered data only with a score of 8.5 or higher and used it to finetune KoRAE for better performance. As a result, we were able to filter the dataset 64k to 12k! This 'KoRAE_filtered_12k' is the result of all previous processes.

Github Repository

For the more specific information, please check the following Repository

Citation

Thanks to @kyujinpy and @nlp-ai for providing Korean datasets.

• KO-Platypus
• Korean-OpenOrca

@inproceedings{lee2023kullm,
  title={KULLM: Learning to Construct Korean Instruction-following Large Language Models},
  author={Lee, SeungJun and Lee, Taemin and Lee, Jeongwoo and Jang, Yoona and Lim, Heuiseok},
  booktitle={Annual Conference on Human and Language Technology},
  pages={196--202},
  year={2023},
  organization={Human and Language Technology}
}

@misc{chen2023alpagasus,
      title={AlpaGasus: Training A Better Alpaca with Fewer Data}, 
      author={Lichang Chen and Shiyang Li and Jun Yan and Hai Wang and Kalpa Gunaratna and Vikas Yadav and Zheng Tang and Vijay Srinivasan and Tianyi Zhou and Heng Huang and Hongxia Jin},
      year={2023},
      eprint={2307.08701},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}