ORPO-DPO-mix-40k v1.2

This dataset is designed for ORPO or DPO training. See Fine-tune Llama 3 with ORPO for more information about how to use it. It is a combination of the following high-quality DPO datasets:

• argilla/Capybara-Preferences: highly scored chosen answers >=5 (7,424 samples)
• argilla/distilabel-intel-orca-dpo-pairs: highly scored chosen answers >=9, not in GSM8K (2,299 samples)
• argilla/ultrafeedback-binarized-preferences-cleaned: highly scored chosen answers >=5 (22,799 samples)
• argilla/distilabel-math-preference-dpo: highly scored chosen answers >=9 (2,181 samples)
• unalignment/toxic-dpo-v0.2 (541 samples)
• M4-ai/prm_dpo_pairs_cleaned (7,958 samples)
• jondurbin/truthy-dpo-v0.1 (1,016 samples)

Rule-based filtering was applied to remove gptisms in the chosen answers (2,206 samples).

Thanks to argilla, unalignment, M4-ai, and jondurbin for providing the source datasets.

🔎 Usage

v1.2 adds a question column to ensure compatibility with both DPO and ORPO formats in Axolotl.

Here's an example as an ORPO dataset in Axolotl:

rl: orpo
orpo_alpha: 0.1
chat_template: chatml
datasets:
  - path: mlabonne/orpo-dpo-mix-40k
    type: chat_template.argilla
    chat_template: chatml

For DPO, I recommend using mlabonne/orpo-dpo-mix-40k-flat instead.

Toxicity

Note that ORPO-DPO-mix-40k contains a dataset (toxic-dpo-v0.2) designed to prompt the model to answer illegal questions. You can remove it as follows:

dataset = load_dataset('mlabonne/orpo-dpo-mix-40k', split='train')
dataset = dataset.filter(
    lambda r: r["source"] != "toxic-dpo-v0.2"
)

History

I'm saving previous versions of this dataset in different branches.

• v1.0