Swallow Instruct v0.1 Dataset

This dataset was used for supervised fine-tuning (SFT) of the Swallow v0.1 model series.

Model Index

The following Instruct models were created using this dataset:

• Llama-3-Swallow-8B-Instruct-v0.1
• Llama-3-Swallow-70B-Instruct-v0.1
• Swallow-7b-instruct-v0.1
• Swallow-13b-instruct-v0.1
• Swallow-70b-instruct-v0.1

Note: The data used for Swallow-MS-7b-instruct-v0.1 is different.

Statistical Information

Data Format

The data is structured as follows:

{
  "conversation": [
    {"role": "user", "content": "USER_MESSAGE1"},
    {"role": "assistant", "content": "ASSISTANT_MESSAGE1"},
    {"role": "user", "content": "USER_MESSAGE2"},
    {"role": "assistant", "content": "ASSISTANT_MESSAGE2"},
    ...
  ]
}

For SFT, it is recommended to calculate loss only on the assistant's responses.

Data Construction Method

oasst2-top1-en

We extracted only the highest-rated dialogues from the OpenAssistant2 conversation tree.

oasst1-21k-ja-imitation_alpha

We used the llm-jp/oasst1-21k-ja dataset, which is a machine translation of OpenAssistant1 (Köpf et al., 2023). The instructions were input into Mixtral-8x7B-Instruct-v0.1 to generate responses. The prompt format follows Mixtral's format. The following parameters were used:

max_length: 4096
top_p: 0.95
temperature: 1.0
repetition_penalty: 1.0
do_sample: True

oasst1-21k-ja-imitation_beta

This dataset uses the same parameters as the alpha version, with one addition: "日本語で応答してください。" (Please respond in Japanese) was appended to the end of each user input.

Authors

Here are the team members:

• From Tokyo Institute of Technology Okazaki Laboratory, the following members:
  • Naoaki Okazaki
  • Sakae Mizuki
  • Youmi Ma
  • Koki Maeda
  • Kakeru Hattori
  • Masanari Ohi
  • Taihei Shiotani
  • Koshiro Saito
• From Tokyo Institute of Technology YOKOTA Laboratory, the following members:
  • Rio Yokota
  • Kazuki Fujii
  • Taishi Nakamura
  • Takumi Okamoto
  • Ishida Shigeki
• From Artificial Intelligence Research Center, AIST, Japan, the following members:
  • Hiroya Takamura

How to Cite

If you find our work helpful, please feel free to cite us.

@misc{llama3swallow,
      title={Llama 3 Swallow},
      url={https://swallow-llm.github.io/llama3-swallow.en.html},
      author={Swallow LLM},
      year={2024},
}