🔥 Chat with Magpie Here!

🐦 Llama-3-8B-Magpie-Align-v0.2

Project Web: https://magpie-align.github.io/

Online Model Demo: https://huggingface.co/spaces/flydust/Chat-with-Magpie

Arxiv Technical Report: https://arxiv.org/abs/2406.08464

Codes: https://github.com/magpie-align/magpie

🧐 About This Model

This model is an aligned version of meta-llama/Meta-Llama-3-8B. We apply the following pipeline:

• We first use Magpie-Align/Magpie-Pro-MT-300K-v0.1 and Magpie-Align/Magpie-Reasoning-150K dataset and perform SFT -> Magpie-Align/Llama-3-8B-Magpie-Align-SFT-v0.2
• We then perform DPO on the princeton-nlp/llama3-ultrafeedback-armorm dataset.

The overall performance is even better than the official Llama-3-8B-Instruct Model!

• Alpaca Eval 2 (vs GPT-4-Turbo-1106): 49.86 (LC), 51.98 (WR)
• Alpaca Eval 2 (vs Llama-3-8B-Instruct): 75.17 (LC), 78.20 (WR)
• Arena Hard: 37.5
• WildBench WB-Score: 42.7
• Zero-Eval GSM: 46.70

🔥 Model Performance

We compare our Llama-3-8B-Magpie-Align with official and other open-aligned LLMs that have been fine-tuned from base models and have publicly released their training datasets. The results are as follows:

+---------------------------------------------+--------------------+--------------------+-----------------------+------------+
|               Aligned Model ID              |      MT-Bench      |   Alpaca Eval 2    |     Alpaca Eval 2     | Arena Hard |
|                                             |                    | (GPT-4-Turbo-1106) | (Llama-3-8B-Instruct) |            |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
|                                             |  R1  |  R2  |  AVG |   LC WR  |    WR   |   LC WR   |     WR    |    Score   |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| meta-llama/Meta-Llama-3-8B-Instruct         | 8.31 | 7.65 | 7.98 |   22.92  |  22.57  |     50    |     50    |    20.6    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| princeton-nlp/Llama-3-Base-8B-SFT-DPO       | 8.12 | 7.23 | 7.67 |   17.71  |  15.34  |   43.73   |   38.80   |    14.8    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| NousResearch/Hermes-2-Pro-Llama-3-8B        | 8.05 | 7.35 | 7.70 |   15.60  |  12.86  |   36.37   |   30.52   |    11.5    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| allenai/llama-3-tulu-2-dpo-8b               | 7.71 | 7.15 | 7.43 |   14.89  |  14.80  |   35.43   |   35.42   |    11.7    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| cognitivecomputations/dolphin-2.9-llama3-8b | 7.97 | 6.98 | 7.47 |   12.50  |   8.79  |   32.67   |   22.80   |     8.2    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| openchat/openchat-3.6-8b-20240522           | 7.83 | 7.23 | 7.53 |   17.70  |  12.53  |   41.30   |   30.79   |     6.7    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.1   | 8.01 | 7.63 | 7.82 |   38.52  |  38.47  |   69.37   |   70.05   |    32.4    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+
| Magpie-Align/Llama-3-8B-Magpie-Align-v0.2   | 7.81 | 7.64 | 7.73 |   49.86  |  51.98  |   75.17   |   78.20   |    37.5    |
+---------------------------------------------+------+------+------+----------+---------+-----------+-----------+------------+

👀 Other Information

License: Please follow Meta Llama 3 Community License.

Conversation Template: Please use Llama 3 official chat template for the best performance.

How to use it? Please check the official Llama 3 repository for detailed instructions. Simply replace the original model_id with Magpie-Align/Llama-3-8B-Magpie-Align-SFT-v1.0.

The detailed training pipeline is as follows.

Stage 1: Supervised Fine-tuning

We use Axolotl for SFT.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 79
• num_epochs: 2

Training results

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1

Internal name for identification: Llama-3-8B-Magpie-Mix-300KMT-150KR. Please change the model name in the below Axolotl config.

base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: Magpie-Align/Magpie-Reasoning-150K
    type: sharegpt
    conversation: llama3
  - path: Magpie-Align/Magpie-Pro-MT-300K-v0.1
    type: sharegpt
    conversation: llama3
dataset_prepared_path: last_run_prepared
val_set_size: 0.001
output_dir: axolotl_out/Llama-3-8B-Magpie-Mix-300KMT-150KR

sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project: SynDa
wandb_entity:
wandb_watch:
wandb_name: Llama-3-8B-Magpie-Mix-300KMT-150KR
wandb_log_model:
hub_model_id: Magpie-Align/Llama-3-8B-Magpie-Mix-300KMT-150KR

gradient_accumulation_steps: 32
micro_batch_size: 1
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_ratio: 0.1
evals_per_epoch: 5
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|end_of_text|>

Stage 2: Direct Preference Optimization

We use alignment handbook for DPO.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-07
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1

# Customized Configs
model_name_or_path: Magpie-Align/Llama-3-8B-Magpie-Mix-300KMT-150KR
hub_model_id: Magpie-Align/Llama-3-8B-Magpie-Align-v0.2-RC
output_dir: alignment_handbook_out/Llama-3-8B-Magpie-Align-v0.2-RC
run_name: Llama-3-8B-Magpie-Align-v0.2-RC

dataset_mixer:
  princeton-nlp/llama3-ultrafeedback-armorm: 1.0
dataset_splits:
- train
- test
preprocessing_num_workers: 24

# DPOTrainer arguments
bf16: true
beta: 0.01
learning_rate: 0.8e-6
gradient_accumulation_steps: 8
per_device_train_batch_size: 2
per_device_eval_batch_size: 4
num_train_epochs: 1
max_length: 2048
max_prompt_length: 1800
warmup_ratio: 0.1
logging_steps: 1
lr_scheduler_type: cosine
optim: adamw_torch

torch_dtype: null
use_flash_attention_2: true
do_eval: true
evaluation_strategy: steps
eval_steps: 100
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: False
log_level: info
push_to_hub: true
save_strategy: "steps"
save_steps: 100
save_total_limit: 1
seed: 42
report_to:
- wandb

Downstream Performance

Paper Abstract

📚 Citation

If you find the model, data, or code useful, please cite our paper:

@article{xu2024magpie,
	title={Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing}, 
	author={Zhangchen Xu and Fengqing Jiang and Luyao Niu and Yuntian Deng and Radha Poovendran and Yejin Choi and Bill Yuchen Lin},
	year={2024},
	eprint={2406.08464},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
}

Please also cite the creators of preference datasets:

SimPO paper:

@article{meng2024simpo,
  title={{SimPO}: Simple preference optimization with a reference-free reward},
  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
  journal={arXiv preprint arXiv:2405.14734},
  year={2024}
}

UltraFeedback paper:

@article{cui2023ultrafeedback,
  title={{UltraFeedback}: Boosting language models with high-quality feedback},
  author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
  journal={arXiv preprint arXiv:2310.01377},
  year={2023}
}

ArmoRM paper:

@article{wang2024interpretable,
  title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
  author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
  journal={arXiv preprint arXiv:2406.12845},
  year={2024}
}

Questions? Please contact Zhangchen by email.