Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

📢 A SMALLER AFFORDABLE MoE MODEL FOR EVERYONE!!

🤗 Model Weights | 🚀 Quick Start | ⚙️ Installation Guide | 🚧 Expert Construction | 💬 Supervised Fine-Tuning (SFT) | 💎 Evaluation
📃 Technical Report

🚀 QuickStart

⚙️ Installation

1. Prepare conda environment: conda create -n smoe python=3.11 (If your environment name is not smoe, you may need to change environment in launching scripts)
2. Add correct environment variables in ~/.bashrc (gcc is set to newer version for installing flash-attn). e.g.: (如果服务器上有cuda驱动，不用这一步安装和写PATH)

   export PATH=/home/zll/tools/cuda-11.8/bin:$PATH
   export LD_LIBRARY_PATH=/home/zll/tools/cuda-11.8/lib64:$LD_LIBRARY_PATH
   export PATH=/home/zll/tools/gcc-10.1.0/bin:$PATH
   export LD_LIBRARY_PATH=/home/zll/tools/gcc-10.1.0/lib64:$LD_LIBRARY_PATH
   

3. Take the variables into effect: source ~/.bashrc
4. Install PyTorch (CUDA-11.8): pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
5. Install dependencies （requirements里的pkgs，确保transformers和flash-attn的版本，其他版本可以不一致）: pip install -r requirements.txt
6. Install flash-attn: pip install flash-attn==2.6.1 --no-build-isolation. You may need to follow the flash-attn installation instructions to avoid some errors.
7. Install the latest Git: conda install git
8. Clone the repo: [email protected]: CaitlinZOO/moe_mt.git (If you don't setup the ssh key to GitHub, you may not able to clone through ssh. Check the docs about it.)
9. Change current directory: cd LLaMA-MoE-v2
10. Install smoe in editable mode: pip install -e .[dev]
11. Setup pre-commit hooks （可以不用设置）: pre-commit install

（先转换复制model）

• LLaMA-MoE 2group 4experts: bash scripts/expert_construction/convert/convert_mixtral_2group_base.sh

For more information, please refer to Expert Construction docs.

💬 Supervised Fine-Tuning (SFT)

• 训练参数

  python train.py \
  --dataset_save_dir=${data_dir}    数据处理后保存的路径，如果已经有处理好的，直接加载
  --manifest_files=${dataset_dir_or_path}     数据全路径文件，多个文件用 | 隔开，目前只能是json或者csv文件
  --input_fields="src_text|src_text"     数据中指定的字段，作为gpt的输入
  --output_fields="src_text|tgt_text"     数据中指定的字段，作为gpt的生成
  --instructions="|"     指令，一个数据文件对应一个，用 | 隔开， lm任务是空，没有指令，翻译任务比如是  "Please translate the English text into Spanish: | Please     translate the English text into French: "
    
  

• sft stage_1 lm : bash scripts/sft/sft_lm_2group_4e_top1_base.sh

• sft stage_2 st : bash scripts/sft/sft_mt_2group_4e_top1_base.sh

• NOTICE: Please create logs/ folder manually: mkdir -p logs

  We provide simple examples of SFT to build chatbots. Please refer to SFT docs for more details.

📑 Citation

@misc{llama-moe-v2,
  title={LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training},
  author={Xiaoye Qu, Daize Dong, Xuyang Hu, Tong Zhu, Weigao Sun, Yu Cheng},
  year={2024},
  month={Nov},
  url={https://arxiv.org/abs/2411.15708}
}

LLaMA-MoE Team w/ ❤️