Chat Templates for 🤗 HuggingFace Large Language Models
Jinja
714
83 commits
updated Dec 13, 2024
NOTICE: I may not update or maintain this repository actively, as I noticed that recent HF LLMs have usually implemented their chat templates in the tokenizer_config.json file. I am glad about this common practice and it will definitely facilitate out-of-box use of these powerful models.
This is a repository that includes proper chat templates (or input formats) for instruction-tuned large language models (LLMs), to support transformers's chat_template feature. If you are interested to include more chat templates, feel free to open a pull request
If you find this repo useful, please kindly cite it:
@misc{zheng-2024-chat-templates,
author = {Zheng, Chujie},
title = {Chat Templates for HuggingFace Large Language Models},
year = {2024},
howpublished = {\url{https://github.com/chujiezheng/chat_templates}}
}
chat_templates contains the jinja files of collected chat templates, which can be directly replaced in the Huggingface tokenizers
generation_configs contains the corresponding json configs used for controlling the ending of response generations. Specially, the stop_token_ids should be directly passed into the generate method by the eos_token_id argument
Important NOTE: As mentioned in this issue, the messages should contain at least one user message. It is strongly not recommented to pass only the system message, as there may result in unexpected outputs (because the models are not trained in this way).
This example may check if the jinja file is correctly implemented.
from transformers import AutoTokenizer
toker = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", token="YOUR_OWN_TOKEN")
messages = [
{'role': 'system', 'content': 'This is a system prompt.'},
{'role': 'user', 'content': 'This is the first user input.'},
{'role': 'assistant', 'content': 'This is the first assistant response.'},
{'role': 'user', 'content': 'This is the second user input.'},
]
print('###### Default (yet Correct) Chat Template ######')
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
print('###### Corrected Chat Template ######')
chat_template = open('./chat_templates/llama-3-instruct.jinja').read()
chat_template = chat_template.replace(' ', '').replace('\n', '')
toker.chat_template = chat_template
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
Expected output:
###### Default (yet Correct) Chat Template ######
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the first user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
This is the first assistant response.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the second user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
###### Corrected Chat Template ######
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the first user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
This is the first assistant response.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the second user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
For mistral-instruct (also gemma-it), it does not natively support the system message, so passing the system message would raise error.
from transformers import AutoTokenizer
toker = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
messages = [
{'role': 'system', 'content': 'This is a system prompt.'},
{'role': 'user', 'content': 'This is the first user input.'},
{'role': 'assistant', 'content': 'This is the first assistant response.'},
{'role': 'user', 'content': 'This is the second user input.'},
]
print('###### Default (but Improper) Chat Template ######')
# raising error
#print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
print('###### Corrected Chat Template ######')
chat_template = open('./chat_templates/mistral-instruct.jinja').read()
chat_template = chat_template.replace(' ', '').replace('\n', '')
toker.chat_template = chat_template
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
Expected output:
###### Default (but Error-Raising) Chat Template ######
jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...
###### Corrected Chat Template ######
<s>[INST] This is a system prompt.
This is the first user input. [/INST] This is the first assistant response. </s>[INST] This is the second user input. [/INST]
NOTE: In fast-chat, vicuna does not add linebreaks between roles' messages. But I found that adding linebreaks leads to a bit better performance (especially for the v1.5 version).
Also, I found vicuna-7/13/33b-v1.3 may not work well when given a system message different from its default one. So I would recommend to use vicuna-7/13b-v1.5 instead.
from transformers import AutoTokenizer
toker = AutoTokenizer.from_pretrained("lmsys/vicuna-7b-v1.5")
messages = [
{'role': 'system', 'content': 'This is a system prompt.'},
{'role': 'user', 'content': 'This is the first user input.'},
{'role': 'assistant', 'content': 'This is the first assistant response.'},
{'role': 'user', 'content': 'This is the second user input.'},
]
print('###### Default (but Improper) Chat Template ######')
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
print('###### Corrected Chat Template ######')
chat_template = open('./chat_templates/vicuna.jinja').read()
chat_template = chat_template.replace(' ', '').replace('\n', '')
toker.chat_template = chat_template
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
Expected output:
###### Default (but Improper) Chat Template ######
<s>[INST] <<SYS>>
This is a system prompt.
<</SYS>>
This is the first user input. [/INST] This is the first assistant response. </s><s>[INST] This is the second user input. [/INST]
###### Corrected Chat Template ######
<s>This is a system prompt.
USER: This is the first user input.
ASSISTANT: This is the first assistant response.</s>
USER: This is the second user input.
ASSISTANT:
NOTE: The listed models are not inclusive and also include other-sized ones in the same model family
ibm-granite/granite-3.0-2b-instruct, ibm-granite/granite-3.0-8b-instruct, ibm-granite/granite-3.0-1b-a400m-instruct, ibm-granite/granite-3.0-3b-a800m-instructchat_templates/granite-3.0-instruct.jinjageneration_configs/granite-3.0-instruct.jsonmeta-llama/Meta-Llama-3.2-1B-Instruct, meta-llama/Meta-Llama-3.2-3B-Instructchat_templates/llama-3-instruct.jinjageneration_configs/llama-3.1-instruct.jsonmeta-llama/Meta-Llama-3.1-8B-Instruct, meta-llama/Meta-Llama-3.1-405B-Instruct-FP8chat_templates/llama-3-instruct.jinjageneration_configs/llama-3.1-instruct.jsonmeta-llama/Meta-Llama-3-8B-Instructchat_templates/llama-3-instruct.jinjageneration_configs/llama-3-instruct.jsonmeta-llama/Llama-2-7b-chat-hf, meta-llama/CodeLlama-7b-Instruct-hfchat_templates/llama-2-chat.jinjageneration_configs/llama-2-chat.jsonQwen/Qwen2-7B-Instruct, Qwen/Qwen1.5-7B-Chatchat_templates/chatml.jinjageneration_configs/qwen2-instruct.jsonmistralai/Mistral-7B-Instruct-v0.3, mistralai/Mixtral-8x7B-Instruct-v0.1chat_templates/mistral-instruct.jinjageneration_configs/mistral-instruct.jsonmicrosoft/Phi-3-mini-4k-instruct, microsoft/Phi-3-small-8k-instructchat_templates/phi-3.jinja, chat_templates/phi-3-small.jinjageneration_configs/phi-3.json, generation_configs/phi-3-small.jsonPhi-3-mini/medium and Phi-3-small adopt different configs01-ai/Yi-1.5-6B-Chat, 01-ai/Yi-6B-Chatchat_templates/chatml.jinjageneration_configs/yi-chat.jsongoogle/gemma-7b-it, google/gemma-2-9b-itchat_templates/gemma-it.jinjageneration_configs/gemma-it.jsonnvidia/Llama3-ChatQA-1.5-8Bchat_templates/chatqa.jinjageneration_configs/chatqa.jsonopenchat/openchat_3.5, berkeley-nest/Starling-LM-7B-alphachat_templates/openchat-3.5.jinjageneration_configs/openchat-3.5.jsonzephyr-7b-alphachat_templates/zephyr.jinjageneration_configs/zephyr.jsonvicuna-7b-v1.5, vicuna-7b-v1.3chat_templates/vicuna.jinjageneration_configs/vicuna.jsonmicrosoft/Orca-2-7bchat_templates/chatml.jinjageneration_configs/orca-2.jsontiiuae/falcon-7b-instructchat_templates/falcon-instruct.jinjaupstage/SOLAR-10.7B-Instruct-v1.0chat_templates/solar-instruct.jinjageneration_configs/solar-instruct.jsontatsu-lab/alpaca-7b-wdiffchat_templates/alpaca.jinjageneration_configs/alpaca.jsonLLM360/AmberChat, LLM360/AmberSafechat_templates/amberchat.jinjageneration_configs/amberchat.jsonIlyaGusev/saiga_mistral_7b_lorachat_templates/saiga.jinjageneration_configs/saiga.json(top 24 of 26)
1,113 followers · starred May 2024
937 followers · starred May 2024
701 followers · starred Jan 2025
463 followers · starred Feb 2024
Chat Templates for 🤗 HuggingFace Large Language Models
Jinja
714
83 commits
updated Dec 13, 2024
NOTICE: I may not update or maintain this repository actively, as I noticed that recent HF LLMs have usually implemented their chat templates in the tokenizer_config.json file. I am glad about this common practice and it will definitely facilitate out-of-box use of these powerful models.
This is a repository that includes proper chat templates (or input formats) for instruction-tuned large language models (LLMs), to support transformers's chat_template feature. If you are interested to include more chat templates, feel free to open a pull request
If you find this repo useful, please kindly cite it:
@misc{zheng-2024-chat-templates,
author = {Zheng, Chujie},
title = {Chat Templates for HuggingFace Large Language Models},
year = {2024},
howpublished = {\url{https://github.com/chujiezheng/chat_templates}}
}
chat_templates contains the jinja files of collected chat templates, which can be directly replaced in the Huggingface tokenizers
generation_configs contains the corresponding json configs used for controlling the ending of response generations. Specially, the stop_token_ids should be directly passed into the generate method by the eos_token_id argument
Important NOTE: As mentioned in this issue, the messages should contain at least one user message. It is strongly not recommented to pass only the system message, as there may result in unexpected outputs (because the models are not trained in this way).
This example may check if the jinja file is correctly implemented.
from transformers import AutoTokenizer
toker = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", token="YOUR_OWN_TOKEN")
messages = [
{'role': 'system', 'content': 'This is a system prompt.'},
{'role': 'user', 'content': 'This is the first user input.'},
{'role': 'assistant', 'content': 'This is the first assistant response.'},
{'role': 'user', 'content': 'This is the second user input.'},
]
print('###### Default (yet Correct) Chat Template ######')
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
print('###### Corrected Chat Template ######')
chat_template = open('./chat_templates/llama-3-instruct.jinja').read()
chat_template = chat_template.replace(' ', '').replace('\n', '')
toker.chat_template = chat_template
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
Expected output:
###### Default (yet Correct) Chat Template ######
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the first user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
This is the first assistant response.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the second user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
###### Corrected Chat Template ######
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the first user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
This is the first assistant response.<|eot_id|><|start_header_id|>user<|end_header_id|>
This is the second user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
For mistral-instruct (also gemma-it), it does not natively support the system message, so passing the system message would raise error.
from transformers import AutoTokenizer
toker = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
messages = [
{'role': 'system', 'content': 'This is a system prompt.'},
{'role': 'user', 'content': 'This is the first user input.'},
{'role': 'assistant', 'content': 'This is the first assistant response.'},
{'role': 'user', 'content': 'This is the second user input.'},
]
print('###### Default (but Improper) Chat Template ######')
# raising error
#print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
print('###### Corrected Chat Template ######')
chat_template = open('./chat_templates/mistral-instruct.jinja').read()
chat_template = chat_template.replace(' ', '').replace('\n', '')
toker.chat_template = chat_template
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
Expected output:
###### Default (but Error-Raising) Chat Template ######
jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...
###### Corrected Chat Template ######
<s>[INST] This is a system prompt.
This is the first user input. [/INST] This is the first assistant response. </s>[INST] This is the second user input. [/INST]
NOTE: In fast-chat, vicuna does not add linebreaks between roles' messages. But I found that adding linebreaks leads to a bit better performance (especially for the v1.5 version).
Also, I found vicuna-7/13/33b-v1.3 may not work well when given a system message different from its default one. So I would recommend to use vicuna-7/13b-v1.5 instead.
from transformers import AutoTokenizer
toker = AutoTokenizer.from_pretrained("lmsys/vicuna-7b-v1.5")
messages = [
{'role': 'system', 'content': 'This is a system prompt.'},
{'role': 'user', 'content': 'This is the first user input.'},
{'role': 'assistant', 'content': 'This is the first assistant response.'},
{'role': 'user', 'content': 'This is the second user input.'},
]
print('###### Default (but Improper) Chat Template ######')
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
print('###### Corrected Chat Template ######')
chat_template = open('./chat_templates/vicuna.jinja').read()
chat_template = chat_template.replace(' ', '').replace('\n', '')
toker.chat_template = chat_template
print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
Expected output:
###### Default (but Improper) Chat Template ######
<s>[INST] <<SYS>>
This is a system prompt.
<</SYS>>
This is the first user input. [/INST] This is the first assistant response. </s><s>[INST] This is the second user input. [/INST]
###### Corrected Chat Template ######
<s>This is a system prompt.
USER: This is the first user input.
ASSISTANT: This is the first assistant response.</s>
USER: This is the second user input.
ASSISTANT:
NOTE: The listed models are not inclusive and also include other-sized ones in the same model family
ibm-granite/granite-3.0-2b-instruct, ibm-granite/granite-3.0-8b-instruct, ibm-granite/granite-3.0-1b-a400m-instruct, ibm-granite/granite-3.0-3b-a800m-instructchat_templates/granite-3.0-instruct.jinjageneration_configs/granite-3.0-instruct.jsonmeta-llama/Meta-Llama-3.2-1B-Instruct, meta-llama/Meta-Llama-3.2-3B-Instructchat_templates/llama-3-instruct.jinjageneration_configs/llama-3.1-instruct.jsonmeta-llama/Meta-Llama-3.1-8B-Instruct, meta-llama/Meta-Llama-3.1-405B-Instruct-FP8chat_templates/llama-3-instruct.jinjageneration_configs/llama-3.1-instruct.jsonmeta-llama/Meta-Llama-3-8B-Instructchat_templates/llama-3-instruct.jinjageneration_configs/llama-3-instruct.jsonmeta-llama/Llama-2-7b-chat-hf, meta-llama/CodeLlama-7b-Instruct-hfchat_templates/llama-2-chat.jinjageneration_configs/llama-2-chat.jsonQwen/Qwen2-7B-Instruct, Qwen/Qwen1.5-7B-Chatchat_templates/chatml.jinjageneration_configs/qwen2-instruct.jsonmistralai/Mistral-7B-Instruct-v0.3, mistralai/Mixtral-8x7B-Instruct-v0.1chat_templates/mistral-instruct.jinjageneration_configs/mistral-instruct.jsonmicrosoft/Phi-3-mini-4k-instruct, microsoft/Phi-3-small-8k-instructchat_templates/phi-3.jinja, chat_templates/phi-3-small.jinjageneration_configs/phi-3.json, generation_configs/phi-3-small.jsonPhi-3-mini/medium and Phi-3-small adopt different configs01-ai/Yi-1.5-6B-Chat, 01-ai/Yi-6B-Chatchat_templates/chatml.jinjageneration_configs/yi-chat.jsongoogle/gemma-7b-it, google/gemma-2-9b-itchat_templates/gemma-it.jinjageneration_configs/gemma-it.jsonnvidia/Llama3-ChatQA-1.5-8Bchat_templates/chatqa.jinjageneration_configs/chatqa.jsonopenchat/openchat_3.5, berkeley-nest/Starling-LM-7B-alphachat_templates/openchat-3.5.jinjageneration_configs/openchat-3.5.jsonzephyr-7b-alphachat_templates/zephyr.jinjageneration_configs/zephyr.jsonvicuna-7b-v1.5, vicuna-7b-v1.3chat_templates/vicuna.jinjageneration_configs/vicuna.jsonmicrosoft/Orca-2-7bchat_templates/chatml.jinjageneration_configs/orca-2.jsontiiuae/falcon-7b-instructchat_templates/falcon-instruct.jinjaupstage/SOLAR-10.7B-Instruct-v1.0chat_templates/solar-instruct.jinjageneration_configs/solar-instruct.jsontatsu-lab/alpaca-7b-wdiffchat_templates/alpaca.jinjageneration_configs/alpaca.jsonLLM360/AmberChat, LLM360/AmberSafechat_templates/amberchat.jinjageneration_configs/amberchat.jsonIlyaGusev/saiga_mistral_7b_lorachat_templates/saiga.jinjageneration_configs/saiga.json(top 24 of 26)
1,113 followers · starred May 2024
937 followers · starred May 2024
701 followers · starred Jan 2025
463 followers · starred Feb 2024