maziao/Huggingface-PaddlePaddle

Convert HuggingFace code and pretrained models to a PaddlePaddle supported format.

21

stars

25

commits

Python

primary language

Jun 25, 2024

updated

README

Huggingface-PaddlePaddle

Convert HuggingFace code and pretrained models to a PaddlePaddle supported format.

Supported models

ID	Family	Converted Checkpoints	Article
1	GPT2	gpt2	Language Models are Unsupervised Multitask Learners
2	GPT-Neo	EleutherAI/gpt-neo-125m	GPT-NeoX-20B: An Open-Source Autoregressive Language Model
3	OPT	facebook/opt-125m	OPT: Open Pre-trained Transformer Language Models
4	BLOOM	YeungNLP/bloom-396m-zh	BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
5	LLaMa	TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T	LLaMA: Open and Efficient Foundation Language Models
6	DITTO	Finetuned	Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
7	ScaleGrad	Finetuned	Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation
8	SimCTG	Finetuned	A Contrastive Framework for Neural Text Generation
9	Unlikelihood-Token	Finetuned	Neural Text Generation with Unlikelihood Training
10	Unlikelihood-Seq	Finetuned	Neural Text Generation with Unlikelihood Training
11	Qwen-1.5	Qwen/Qwen1.5-0.5B	Qwen Technical Report
12	GPT-SW3	AI-Sweden-Models/gpt-sw3-126m	GPT-SW3: An Autoregressive Language Model for the Nordic Languages
13	Galactica	facebook/galactica-125m	Galactica: A Large Language Model for Science
14	DeepSeek LLM	deepseek-ai/deepseek-coder-1.3b-base	DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
15	InternLM2	internlm/internlm2-1_8b	InternLM - GitHub Repo
16	Pythia	EleutherAI/pythia-70m	Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
17	Phi-1.5	microsoft/phi-1_5	Textbooks Are All You Need II: phi-1.5 technical report

Procedures

Step 1. Download HuggingFace model checkpoints

huggingface-cli download --resume-download PRETRAINED_MODEL_NAME --cache-dir CACHE_DIR

Step 2. Transform HuggingFace checkpoints (PyTorch) to PaddlePaddle

python transform_checkpoint/transform_xxx.py --hf-repo CACHE_DIR --pd-repo TARGET_DIR

[Optional] Step 3. Check correctness of transformation

python check_correctness.py --hf-repo CACHE_DIR --pd-repo TARGET_DIR

Step 4. Generate new config file

python generate_config.py --src-config PATH_TO_SRC_CONFIG --mode-name MODEL_NAME --tgt-dir TARGET_DIR

Step 5. Continue training from pretrained checkpoints

CUDA_VISIBLE_DEVICES={x} python train.py \
    --model-config MODEL_CONFIG \
    --model-name MODEL_NAME \
    --tokenizer TOKENIZER \
    --dataset DATASET \
    --criterion CRITERION \
    --pretrained-model-path TARGET_DIR \
    --save-dir SAVE_DIR

Step 6. Evaluation

CUDA_VISIBLE_DEVICES={x} python test.py --dataset DATASET --pretrained-model-path SAVE_DIR

Contributors

25 commits

Similar repos

Tongjilibo/gpt2-ml_15g_corpus

0

PaddlePaddle/PaddleFormers

PaddleFormers is an easy-to-use library of pre-trained large language model zoo based on…

12,988

PaddlePaddle/PaddleNLP

Easy-to-use and powerful LLM and SLM library with awesome model zoo.

12,973

ylwt/PaddlePaddle-TinyLlama-1.1B-intermediate-step-1195k-token-2.5T

0

TheBloke/Llama-2-70B-fp16

47

Feilong607/LLaVA-OneVision-2-Megatron-Bridge

1

Tongjilibo/uie-base

0

ylwt/PaddlePaddle-gpt-neo-125m

0

Languages

Python

95.2%

Shell

4.8%

maziao/Huggingface-PaddlePaddle

Convert HuggingFace code and pretrained models to a PaddlePaddle supported format.

21

stars

25

commits

Python

primary language

Jun 25, 2024

updated

README

Huggingface-PaddlePaddle

Convert HuggingFace code and pretrained models to a PaddlePaddle supported format.

Supported models

ID	Family	Converted Checkpoints	Article
1	GPT2	gpt2	Language Models are Unsupervised Multitask Learners
2	GPT-Neo	EleutherAI/gpt-neo-125m	GPT-NeoX-20B: An Open-Source Autoregressive Language Model
3	OPT	facebook/opt-125m	OPT: Open Pre-trained Transformer Language Models
4	BLOOM	YeungNLP/bloom-396m-zh	BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
5	LLaMa	TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T	LLaMA: Open and Efficient Foundation Language Models
6	DITTO	Finetuned	Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
7	ScaleGrad	Finetuned	Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation
8	SimCTG	Finetuned	A Contrastive Framework for Neural Text Generation
9	Unlikelihood-Token	Finetuned	Neural Text Generation with Unlikelihood Training
10	Unlikelihood-Seq	Finetuned	Neural Text Generation with Unlikelihood Training
11	Qwen-1.5	Qwen/Qwen1.5-0.5B	Qwen Technical Report
12	GPT-SW3	AI-Sweden-Models/gpt-sw3-126m	GPT-SW3: An Autoregressive Language Model for the Nordic Languages
13	Galactica	facebook/galactica-125m	Galactica: A Large Language Model for Science
14	DeepSeek LLM	deepseek-ai/deepseek-coder-1.3b-base	DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
15	InternLM2	internlm/internlm2-1_8b	InternLM - GitHub Repo
16	Pythia	EleutherAI/pythia-70m	Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
17	Phi-1.5	microsoft/phi-1_5	Textbooks Are All You Need II: phi-1.5 technical report

Procedures

Step 1. Download HuggingFace model checkpoints

huggingface-cli download --resume-download PRETRAINED_MODEL_NAME --cache-dir CACHE_DIR

Step 2. Transform HuggingFace checkpoints (PyTorch) to PaddlePaddle

python transform_checkpoint/transform_xxx.py --hf-repo CACHE_DIR --pd-repo TARGET_DIR

[Optional] Step 3. Check correctness of transformation

python check_correctness.py --hf-repo CACHE_DIR --pd-repo TARGET_DIR

Step 4. Generate new config file

python generate_config.py --src-config PATH_TO_SRC_CONFIG --mode-name MODEL_NAME --tgt-dir TARGET_DIR

Step 5. Continue training from pretrained checkpoints

CUDA_VISIBLE_DEVICES={x} python train.py \
    --model-config MODEL_CONFIG \
    --model-name MODEL_NAME \
    --tokenizer TOKENIZER \
    --dataset DATASET \
    --criterion CRITERION \
    --pretrained-model-path TARGET_DIR \
    --save-dir SAVE_DIR

Step 6. Evaluation

CUDA_VISIBLE_DEVICES={x} python test.py --dataset DATASET --pretrained-model-path SAVE_DIR

Similar repos

Tongjilibo/gpt2-ml_15g_corpus

0

PaddlePaddle/PaddleFormers

PaddleFormers is an easy-to-use library of pre-trained large language model zoo based on…

12,988

PaddlePaddle/PaddleNLP

Easy-to-use and powerful LLM and SLM library with awesome model zoo.

12,973

ylwt/PaddlePaddle-TinyLlama-1.1B-intermediate-step-1195k-token-2.5T

0

TheBloke/Llama-2-70B-fp16

47

Feilong607/LLaVA-OneVision-2-Megatron-Bridge

1

Tongjilibo/uie-base

0

ylwt/PaddlePaddle-gpt-neo-125m

0

Contributors

25 commits

Languages

Python

95.2%

Shell

4.8%