Ongoing research training Mixture of Expert models.
See the codeThis repository is not active. moe-recipes is now actively being developed in other repository.
To install the package, run the following command:
pip install -r requirements.txt
If you want to use the library in multi-nodes, you need to install the below packages:
module load openmpi/4.x.x
pip install mpi4py
To install the FlashAttention, run the following command: (GPU is required)
pip install ninja packaging wheel
pip install flash-attn --no-build-isolation
156 followers · starred May 2024
Ongoing research training Mixture of Expert models.
See the codeThis repository is not active. moe-recipes is now actively being developed in other repository.
To install the package, run the following command:
pip install -r requirements.txt
If you want to use the library in multi-nodes, you need to install the below packages:
module load openmpi/4.x.x
pip install mpi4py
To install the FlashAttention, run the following command: (GPU is required)
pip install ninja packaging wheel
pip install flash-attn --no-build-isolation
156 followers · starred May 2024