COIG Prompt Collection

License

Default Licensing for Sub-Datasets Without Specific License Declaration: In instances where sub-datasets within the COIG-PC Dataset do not have a specific license declaration, the Apache License 2.0 (Apache-2.0) will be the applicable licensing terms by default.

Precedence of Declared Licensing for Sub-Datasets: For any sub-dataset within the COIG-PC Dataset that has an explicitly declared license, the terms and conditions of the declared license shall take precedence and govern the usage of that particular sub-dataset.

Users and developers utilizing the COIG-PC Dataset must ensure compliance with the licensing terms as outlined above. It is imperative to review and adhere to the specified licensing conditions of each sub-dataset, as they may vary.

What is COIG-PC?

The COIG-PC Dataset is a meticulously curated and comprehensive collection of Chinese tasks and data, designed to facilitate the fine-tuning and optimization of language models for Chinese natural language processing (NLP). The dataset aims to provide researchers and developers with a rich set of resources to improve the capabilities of language models in handling Chinese text, which can be utilized in various fields such as text generation, information extraction, sentiment analysis, machine translation, among others.

COIG-PC-Lite is a subset of COIG-PC with only 200 samples from each task file. If you are looking for COIG-PC, please refer to https://huggingface.co/datasets/BAAI/COIG-PC.

Why COIG-PC?

The COIG-PC Dataset is an invaluable resource for the domain of natural language processing (NLP) for various compelling reasons:

Addressing Language Complexity: Chinese is known for its intricacy, with a vast array of characters and diverse grammatical structures. A specialized dataset like COIG-PC, which is tailored for the Chinese language, is essential to adequately address these complexities during model training.

Comprehensive Data Aggregation: The COIG-PC Dataset is a result of an extensive effort in integrating almost all available Chinese datasets in the market. This comprehensive aggregation makes it one of the most exhaustive collections for Chinese NLP.

Data Deduplication and Normalization: The COIG-PC Dataset underwent rigorous manual processing to eliminate duplicate data and perform normalization. This ensures that the dataset is free from redundancy, and the data is consistent and well-structured, making it more user-friendly and efficient for model training.

Fine-tuning and Optimization: The dataset’s instruction-based phrasing facilitates better fine-tuning and optimization of language models. This structure allows models to better understand and execute tasks, which is particularly beneficial in improving performance on unseen or novel tasks.

The COIG-PC Dataset, with its comprehensive aggregation, meticulous selection, deduplication, and normalization of data, stands as an unmatched resource for training and optimizing language models tailored for the Chinese language and culture. It addresses the unique challenges of Chinese language processing and serves as a catalyst for advancements in Chinese NLP.

Who builds COIG-PC?

The bedrock of COIG-PC is anchored in the dataset furnished by stardust.ai, which comprises an aggregation of data collected from the Internet.

And COIG-PC is the result of a collaborative effort involving engineers and experts from over twenty distinguished universities both domestically and internationally. Due to space constraints, it is not feasible to list all of them; however, the following are a few notable institutions among the collaborators:

• Beijing Academy of Artificial Intelligence, China
• Peking University, China
• The Hong Kong University of Science and Technology (HKUST), China
• The University of Waterloo, Canada
• The University of Sheffield, United Kingdom
• Beijing University of Posts and Telecommunications, China
• Multimodal Art Projection
• stardust.ai, China
• LinkSoul.AI, China

For the detailed list of engineers involved in the creation and refinement of COIG-PC, please refer to the paper that will be published subsequently. This paper will provide in-depth information regarding the contributions and the specifics of the dataset’s development process.

How to use COIG-PC?

COIG-PC is structured in a .jsonl file format. Each line in the file represents a single data record and is structured in JSON (JavaScript Object Notation) format. Below is a breakdown of the elements within each line:

instruction: This is a text string that provides the instruction for the task. For example, it might tell the model what to do with the input data.

input: This is the input data that the model needs to process. In the context of translation, it would be the text that needs to be translated.

output: This contains the expected output data after processing the input. In the context of translation, it would be the translated text.

split: Indicates the official split of the original dataset, which is used to categorize data for different phases of model training and evaluation. It can be 'train', 'test', 'valid', etc.

task_type: Contains major and minor categories for the dataset. Major categories are broader, while minor categories can be more specific subcategories.

domain: Indicates the domain or field to which the data belongs.

other: This field can contain additional information or metadata regarding the data record. If there is no additional information, it may be set to null.

Example

Here is an example of how a line in the COIG-PC dataset might be structured:

{
    "instruction": "请把下面的中文句子翻译成英文",
    "input": "我爱你。",
    "output": "I love you.",
    "split": "train",
    "task_type": {
        "major": ["翻译"],
        "minor": ["翻译", "中译英"]
    },
    "domain": ["通用"],
    "other": null
}

In this example: instruction tells the model to translate the following Chinese sentence into English. input contains the Chinese text "我爱你" which means "I love you". output contains the expected translation in English: "I love you". split indicates that this data record is part of the training set. task_type specifies that the major category is "Translation" and the minor categories are "Translation" and "Chinese to English". domain specifies that this data record belongs to the general domain. other is set to null as there is no additional information for this data record.

Update: Aug. 30, 2023

• v1.2: Delete 31 bad task files. Update 99 task files. Rename 2 task files. Add 3 new task files. COIG-PC now has 3339 tasks in total.
• v1.1: Fix 00040-001-000 and 00050-003-000, ignore 00930 and 01373.
• v1.0: First version for arXiv paper.
• v0.6: Upload 28 new tasks. COIG-PC now has 3367 tasks in total.
• v0.5: Upload 202 new tasks. COIG-PC now has 3339 tasks in total.
• v0.4: Upload 1049 new tasks. COIG-PC now has 3137 tasks in total.
• v0.3: Upload 1139 new tasks. COIG-PC now has 2088 tasks in total.
• v0.2: Upload 422 new tasks. COIG-PC now has 949 tasks in total. Add "TopSamplenumPerTask" split where only "Samplenum" samples are used from each task.
• v0.1: Upload 527 tasks.

COIG-PC Citation

If you want to cite COIG-PC dataset, you could use this:

@misc{zhang2023chinese,
      title={Chinese Open Instruction Generalist: A Preliminary Release}, 
      author={Ge Zhang and Yemin Shi and Ruibo Liu and Ruibin Yuan and Yizhi Li and Siwei Dong and Yu Shu and Zhaoqun Li and Zekun Wang and Chenghua Lin and Wenhao Huang and Jie Fu},
      year={2023},
      eprint={2304.07987},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contact Us

To contact us feel free to create an Issue in this repository.