README
LLM-training-sft-dpo-pipeline
SFT Pipeline
-
Prompt templates are in the
templates/instructionsfolder.- Llama3-1 can be used to write the templates
- Question-answering template is in
templates/instructions/question-answering.jinja2
-
Run the following commands to start the container and get a shell into it:
-
Build the Podman image:
podman build -t sft-pipeline-image .or if docker-compose.yml file is present then compose and build the container with the following command:
podman-compose up -d --build sft-pipelineor
docker compose up -d --build sft-pipeline -
Verify the container is running:
podman ps -
Execute a shell in the running container:
podman exec -it -e WANDB_API_KEY=f831d81ce78350019acca3bcd41969beca447b2a -e HF_TOKEN=sss sft-pipeline bashor
docker exec -it sft-pipeline-test bash -c 'export WANDB_API_KEY=f831d81ce78350019acca3bcd41969beca447b2a && export HF_TOKEN=sss && bash' -
Inside the container, run the script to start the training:
python sft_pipeline/main.py
-
-
export NCCL_NVLS_ENABLE=0 -
accelerate launch --config_file=deepspeed_zero2.yaml --gradient_accumulation_steps 8 sft_pipeline/main.py \ --seed 42 \ --packing False \ --output_dir "output_dir" \ --num_train_epochs 3 \ --per_device_train_batch_size 3 \ --gradient_accumulation_steps 2 \ --optim "adamw_torch_fused" \ --logging_steps 10 \ --save_strategy "epoch" \ --learning_rate 2e-4 \ --bf16 \ --tf32 \ --max_grad_norm 0.3 \ --warmup_ratio 0.1 \ --lr_scheduler_type "cosine" \ --report_to wandb \ --log_with wandb \ --wandb_project "SFT" \ --wandb_run_name "topic_class" \ --dataset_batch_size 1000 \ --neftune_noise_alpha 0.1 \ --eval_packing False \ --num_of_sequences 1024 \ --chars_per_token 3.6 \ --max_steps -1 \ --save_steps 500 \ --save_total_limit 3 \ --log_with "wandb" \ --evaluation_strategy "epoch" \ --model_name_or_path "model/Llama-3.2-3B" \ --trust_remote_code \ --model_revision "main" \ --attn_implementation "flash_attention_2" \ --torch_dtype "auto" \ --use_cache \ --lora_task_type "CAUSAL_LM" \ --lora_r 16 \ --lora_target_modules "q_proj" "v_proj" \ --lora_alpha 32 \ --lora_dropout 0.05 \ --use_rslora \ --lora_modules_to_save "embed_tokens" "lm_head" \ --seq_length 2048 \ --log_level "info" \ --dataset_dir "sft_pipeline/data/topic_class_files/" \ --dataset_train_test_split 0.1 \ --task_name "NLP:Topic Classification" \ --bnb_4bit_compute_dtype "float16" \ --bnb_4bit_quant_type "fp4" \ --use_bnb_nested_quant \ --bnb_4bit_quant_storage "float16" \ --load_in_4bit \ --max_sequence_length 4096Starting from preprocessed dataset
-
accelerate launch --config_file=deepspeed_zero2.yaml sft_pipeline/main.py \ --seed 42 \ --packing False \ --output_dir "output_dir/qvac/peft/all_bonito_nlp" \ --num_train_epochs 3 \ --per_device_train_batch_size 3 \ --per_device_eval_batch_size 1 \ --optim "adamw_torch_fused" \ --max_grad_norm 0.3 \ --logging_steps 10 \ --save_strategy "epoch" \ --learning_rate 2e-4 \ --bf16 \ --warmup_ratio 0.1 \ --weight_decay 0.1 \ --lr_scheduler_type "cosine" \ --report_to wandb \ --log_with wandb \ --wandb_project "SFT" \ --wandb_run_name "qvac_all_bonito_nlp" \ --neftune_noise_alpha 0.1 \ --eval_packing False \ --num_of_sequences 128 \ --max_steps -1 \ --save_steps 500 \ --save_total_limit 2 \ --log_with "wandb" \ --evaluation_strategy "epoch" \ --model_name_or_path "model/Llama-3.2-3B-new-tokenizer" \ --trust_remote_code \ --model_revision "main" \ --attn_implementation "flash_attention_2" \ --use_cache \ --lora_task_type "CAUSAL_LM" \ --seq_length 4096 \ --log_level "info" \ --dataset_dir "data/reorganized_data/all_bonito_nlp" \ --use_peft \ --lora_r 8 \ --lora_alpha 32 \ --lora_dropout 0.05 \ --lora_target_modules "qkv_proj" \ --load_preprocessed_dataset \ --load_qvac_modelFine-tuning QVAC model
-
accelerate launch --config_file=deepspeed_zero2.yaml sft_pipeline/main.py \ --seed 42 \ --packing False \ --output_dir "output_dir/qvac/peft/cot" \ --num_train_epochs 3 \ --per_device_train_batch_size 3 \ --per_device_eval_batch_size 1 \ --optim "adamw_torch_fused" \ --max_grad_norm 0.3 \ --logging_steps 10 \ --save_strategy "epoch" \ --learning_rate 2e-4 \ --bf16 \ --warmup_ratio 0.1 \ --weight_decay 0.1 \ --lr_scheduler_type "cosine" \ --report_to wandb \ --log_with wandb \ --wandb_project "SFT" \ --wandb_run_name "qvac_cot" \ --neftune_noise_alpha 0.1 \ --eval_packing False \ --num_of_sequences 512 \ --save_steps 2000 \ --save_total_limit 2 \ --log_with "wandb" \ --evaluation_strategy "epoch" \ --model_name_or_path "model/pytorch_model.bin" \ --trust_remote_code \ --model_revision "main" \ --attn_implementation "flash_attention_2" \ --use_cache \ --lora_task_type "CAUSAL_LM" \ --seq_length 4096 \ --log_level "info" \ --dataset_dir "data/reorganized_data/cot" \ --lora_r 8 \ --lora_alpha 32 \ --lora_dropout 0.05 \ --lora_target_modules ".*qkv_proj" \ --use_peft \ --load_preprocessed_dataset \ --load_qvac_model \ --skip_mergeResume training from a checkpoint
add this argument to the training command:
--resume_from_checkpoint "output_dir/qvac/peft/coding_new/checkpoint-56000" \ -
Task names:
- NLP:Topic Classification
- NLP:Question Answer
- NLP:Summarization
- NLP:Extractive Question Answering
- NLP:Multiple Choice Question Answering
- NLP:Coreference Resolution
- NLP:Paraphrase Generation
- NLP:Natural Language Inference
- NLP:Yes-no Question Answering
- NLP:Sentiment Analysis
- NLP:Textual Entailment
- NLP:Question Answer Open Dedup
- NLP:Few Shot Learning
- COT:Zero Shot Learning
-
Failing tasks:
- NLP:Sentiment Analysis
- NLP:Coreference Resolution
-
References
Similar repos
Languages
Python
95.3%
Jinja
2.1%
Shell
2.0%
README
LLM-training-sft-dpo-pipeline
SFT Pipeline
-
Prompt templates are in the
templates/instructionsfolder.- Llama3-1 can be used to write the templates
- Question-answering template is in
templates/instructions/question-answering.jinja2
-
Run the following commands to start the container and get a shell into it:
-
Build the Podman image:
podman build -t sft-pipeline-image .or if docker-compose.yml file is present then compose and build the container with the following command:
podman-compose up -d --build sft-pipelineor
docker compose up -d --build sft-pipeline -
Verify the container is running:
podman ps -
Execute a shell in the running container:
podman exec -it -e WANDB_API_KEY=f831d81ce78350019acca3bcd41969beca447b2a -e HF_TOKEN=sss sft-pipeline bashor
docker exec -it sft-pipeline-test bash -c 'export WANDB_API_KEY=f831d81ce78350019acca3bcd41969beca447b2a && export HF_TOKEN=sss && bash' -
Inside the container, run the script to start the training:
python sft_pipeline/main.py
-
-
export NCCL_NVLS_ENABLE=0 -
accelerate launch --config_file=deepspeed_zero2.yaml --gradient_accumulation_steps 8 sft_pipeline/main.py \ --seed 42 \ --packing False \ --output_dir "output_dir" \ --num_train_epochs 3 \ --per_device_train_batch_size 3 \ --gradient_accumulation_steps 2 \ --optim "adamw_torch_fused" \ --logging_steps 10 \ --save_strategy "epoch" \ --learning_rate 2e-4 \ --bf16 \ --tf32 \ --max_grad_norm 0.3 \ --warmup_ratio 0.1 \ --lr_scheduler_type "cosine" \ --report_to wandb \ --log_with wandb \ --wandb_project "SFT" \ --wandb_run_name "topic_class" \ --dataset_batch_size 1000 \ --neftune_noise_alpha 0.1 \ --eval_packing False \ --num_of_sequences 1024 \ --chars_per_token 3.6 \ --max_steps -1 \ --save_steps 500 \ --save_total_limit 3 \ --log_with "wandb" \ --evaluation_strategy "epoch" \ --model_name_or_path "model/Llama-3.2-3B" \ --trust_remote_code \ --model_revision "main" \ --attn_implementation "flash_attention_2" \ --torch_dtype "auto" \ --use_cache \ --lora_task_type "CAUSAL_LM" \ --lora_r 16 \ --lora_target_modules "q_proj" "v_proj" \ --lora_alpha 32 \ --lora_dropout 0.05 \ --use_rslora \ --lora_modules_to_save "embed_tokens" "lm_head" \ --seq_length 2048 \ --log_level "info" \ --dataset_dir "sft_pipeline/data/topic_class_files/" \ --dataset_train_test_split 0.1 \ --task_name "NLP:Topic Classification" \ --bnb_4bit_compute_dtype "float16" \ --bnb_4bit_quant_type "fp4" \ --use_bnb_nested_quant \ --bnb_4bit_quant_storage "float16" \ --load_in_4bit \ --max_sequence_length 4096Starting from preprocessed dataset
-
accelerate launch --config_file=deepspeed_zero2.yaml sft_pipeline/main.py \ --seed 42 \ --packing False \ --output_dir "output_dir/qvac/peft/all_bonito_nlp" \ --num_train_epochs 3 \ --per_device_train_batch_size 3 \ --per_device_eval_batch_size 1 \ --optim "adamw_torch_fused" \ --max_grad_norm 0.3 \ --logging_steps 10 \ --save_strategy "epoch" \ --learning_rate 2e-4 \ --bf16 \ --warmup_ratio 0.1 \ --weight_decay 0.1 \ --lr_scheduler_type "cosine" \ --report_to wandb \ --log_with wandb \ --wandb_project "SFT" \ --wandb_run_name "qvac_all_bonito_nlp" \ --neftune_noise_alpha 0.1 \ --eval_packing False \ --num_of_sequences 128 \ --max_steps -1 \ --save_steps 500 \ --save_total_limit 2 \ --log_with "wandb" \ --evaluation_strategy "epoch" \ --model_name_or_path "model/Llama-3.2-3B-new-tokenizer" \ --trust_remote_code \ --model_revision "main" \ --attn_implementation "flash_attention_2" \ --use_cache \ --lora_task_type "CAUSAL_LM" \ --seq_length 4096 \ --log_level "info" \ --dataset_dir "data/reorganized_data/all_bonito_nlp" \ --use_peft \ --lora_r 8 \ --lora_alpha 32 \ --lora_dropout 0.05 \ --lora_target_modules "qkv_proj" \ --load_preprocessed_dataset \ --load_qvac_modelFine-tuning QVAC model
-
accelerate launch --config_file=deepspeed_zero2.yaml sft_pipeline/main.py \ --seed 42 \ --packing False \ --output_dir "output_dir/qvac/peft/cot" \ --num_train_epochs 3 \ --per_device_train_batch_size 3 \ --per_device_eval_batch_size 1 \ --optim "adamw_torch_fused" \ --max_grad_norm 0.3 \ --logging_steps 10 \ --save_strategy "epoch" \ --learning_rate 2e-4 \ --bf16 \ --warmup_ratio 0.1 \ --weight_decay 0.1 \ --lr_scheduler_type "cosine" \ --report_to wandb \ --log_with wandb \ --wandb_project "SFT" \ --wandb_run_name "qvac_cot" \ --neftune_noise_alpha 0.1 \ --eval_packing False \ --num_of_sequences 512 \ --save_steps 2000 \ --save_total_limit 2 \ --log_with "wandb" \ --evaluation_strategy "epoch" \ --model_name_or_path "model/pytorch_model.bin" \ --trust_remote_code \ --model_revision "main" \ --attn_implementation "flash_attention_2" \ --use_cache \ --lora_task_type "CAUSAL_LM" \ --seq_length 4096 \ --log_level "info" \ --dataset_dir "data/reorganized_data/cot" \ --lora_r 8 \ --lora_alpha 32 \ --lora_dropout 0.05 \ --lora_target_modules ".*qkv_proj" \ --use_peft \ --load_preprocessed_dataset \ --load_qvac_model \ --skip_mergeResume training from a checkpoint
add this argument to the training command:
--resume_from_checkpoint "output_dir/qvac/peft/coding_new/checkpoint-56000" \ -
Task names:
- NLP:Topic Classification
- NLP:Question Answer
- NLP:Summarization
- NLP:Extractive Question Answering
- NLP:Multiple Choice Question Answering
- NLP:Coreference Resolution
- NLP:Paraphrase Generation
- NLP:Natural Language Inference
- NLP:Yes-no Question Answering
- NLP:Sentiment Analysis
- NLP:Textual Entailment
- NLP:Question Answer Open Dedup
- NLP:Few Shot Learning
- COT:Zero Shot Learning
-
Failing tasks:
- NLP:Sentiment Analysis
- NLP:Coreference Resolution
-
References
Similar repos
Languages
Python
95.3%
Jinja
2.1%
Shell
2.0%