Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments
1
3 commits
3 linked in READMEs
updated Dec 20, 2024
Authors: Tuka Alhanai [email protected], Adam Kasumovic [email protected], Mohammad Ghassemi [email protected], Aven Zitzelberger [email protected], Jessica Lundin [email protected], Guillaume Chabot-Couture [email protected]
This HuggingFace Dataset contains the human-translated benchmarks we created from our paper, titled as above. Find the paper here: https://arxiv.org/abs/2412.12417
For more information, see the full repository on GitHub: https://github.com/InstituteforDiseaseModeling/Bridging-the-Gap-Low-Resource-African-Languages
Be sure to run pip install datasets to install HuggingFace's datasets package first.
Adjust the top three variables as desired to specify the language, subject, and split of the dataset.
Compared to Winogrande, the MMLU subsets in this dataset have:
from datasets import load_dataset # pip install datasets
from pprint import pprint
from collections import Counter
# TODO: Developer set these three variables as desired
# Afrikaans (af), Amharic (am), Bambara (bm), Igbo (ig), Sepedi (nso), Shona (sn),
# Sesotho (st), Setswana (tn), Tsonga (ts), Xhosa (xh), Zulu (zu)
desired_lang = "af"
# clinical_knowledge, college_medicine, virology
desired_subject = "clinical_knowledge"
# dev, test, val
desired_split = "test"
# Load dataset
dataset_path = "Institute-Disease-Modeling/mmlu-winogrande-afr"
desired_subset = f"mmlu_{desired_subject}_{desired_lang}"
dataset = load_dataset(dataset_path, desired_subset, split=desired_split)
# Inspect Dataset
# General Information
print("\nDataset Features:")
pprint(dataset.features)
print("\nNumber of rows in the dataset:")
print(len(dataset))
# Inspect Questions and Options
# Convert dictionary of lists to list of dictionaries for easier iteration
dataset_list = [dict(zip(dataset[:].keys(), values)) for values in zip(*dataset[:].values())]
print("\nExample Questions and Options:")
for row in dataset_list[:3]: # Inspect the first 3 rows
print(f"Question: {row['Question']}")
print(f"Options: A) {row['OptionA']} | B) {row['OptionB']} | C) {row['OptionC']} | D) {row['OptionD']}")
print(f"Answer: {row['Answer']}")
print("-" * 50)
# Analyze Answer Distribution
answer_distribution = Counter(row['Answer'] for row in dataset)
print("\nAnswer Distribution:")
for answer, count in sorted(answer_distribution.items()):
print(f"Answer {answer}: {count} ({count / len(dataset) * 100:.2f}%)")
# Average Question Length
avg_question_length = sum(len(row['Question']) for row in dataset) / len(dataset)
print(f"\nAverage Question Length: {avg_question_length:.2f} characters")
Be sure to run pip install datasets to install HuggingFace's datasets package first.
Adjust the top two variables as desired to specify the language and split of the dataset.
Compared to MMLU, the Winogrande subsets in this dataset have:
from datasets import load_dataset # pip install datasets
from pprint import pprint
from collections import Counter
# TODO: Developer set these two variables as desired
# Afrikaans (af), Amharic (am), Bambara (bm), Igbo (ig), Sepedi (nso), Shona (sn),
# Sesotho (st), Setswana (tn), Tsonga (ts), Xhosa (xh), Zulu (zu)
desired_lang = "bm"
# dev, test, train_s
desired_split = "train_s"
# Load dataset
dataset_path = "Institute-Disease-Modeling/mmlu-winogrande-afr"
desired_subset = f"winogrande_{desired_lang}"
dataset = load_dataset(dataset_path, desired_subset, split=desired_split)
# Inspect Dataset
# General Information
print("\nDataset Features:")
pprint(dataset.features)
print("\nNumber of rows in the dataset:")
print(len(dataset))
# Inspect Sentences and Options
# Convert dictionary of lists to list of dictionaries for easier iteration
dataset_list = [dict(zip(dataset[:].keys(), values)) for values in zip(*dataset[:].values())]
print("\nExample Sentences and Options:")
for row in dataset_list[:3]: # Inspect the first 3 rows
print(f"Sentence: {row['Sentence']}")
print(f"Options: 1) {row['Option1']} | 2) {row['Option2']}")
print(f"Answer: {row['Answer']}")
print("-" * 50)
# Analyze Answer Distribution
answer_distribution = Counter(row['Answer'] for row in dataset)
print("\nAnswer Distribution:")
for answer, count in sorted(answer_distribution.items()):
print(f"Answer {answer}: {count} ({count / len(dataset) * 100:.2f}%)")
# Average Sentence Length
avg_sentence_length = sum(len(row['Sentence']) for row in dataset) / len(dataset)
print(f"\nAverage Sentence Length: {avg_sentence_length:.2f} characters")
As used in our own experiments, we have prepared fine-tunable versions of the datasets (in GPT format), which are present in the GitHub repository. These datasets can be used with OpenAI's Fine-Tuning API to fine-tune GPT models on our MMLU and Winogrande translations. Note that since MMLU does not have a train set, the entirety of MMLU college medicine is used for training (MMLU college medicine is naturally excluded from testing for fine-tuned models).
Moreover, see here for an example Jupyter Notebook from our GitHub repository that allows the user to fine-tune a number of models by selecting the desired fine-tuning datasets. The notebook then fine-tunes Unsloth's Llama 3 70B IT (the model can be swapped out with similar models) on each fine-tuning dataset and evaluates each fine-tuned model's performance on MMLU and Winogrande test sets (the same as in this HuggingFace Dataset, but formatted into JSONL). Note that using the aforementioned notebook requires a full clone of the GitHub repository and a powerful GPU like a NVIDIA A100 GPU.
For more details, see our paper.
The code in this repository was developed by IDM, the Bill & Melinda Gates Foundation, and Ghamut Corporation to further research in Large Language Models (LLMs) for low-resource African languages by allowing them to be evaluated on question-answering and commonsense reasoning tasks, like those commonly available in English. We’ve made it publicly available under the MIT License to provide others with a better understanding of our research and an opportunity to build upon it for their own work. We make no representations that the code works as intended or that we will provide support, address issues that are found, or accept pull requests. You are welcome to create your own fork and modify the code to suit your own modeling needs as contemplated under the MIT License.
This HuggingFace Dataset includes data derived from the following datasets, each subject to their respective licenses (copied from their respective GitHub repositories):
@article{hendryckstest2021,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}
@article{hendrycks2021ethics,
title={Aligning AI With Shared Human Values},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}
@article{sakaguchi2019winogrande,
title={WinoGrande: An Adversarial Winograd Schema Challenge at Scale},
author={Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin},
journal={arXiv preprint arXiv:1907.10641},
year={2019}
}
Please note that the licenses for the included datasets are separate from and may impose additional restrictions beyond the HuggingFace Dataset's main license.
If you find this HuggingFace Dataset useful, please consider citing it:
@article{,
title={Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments},
author={Tuka Alhanai and Adam Kasumovic and Mohammad Ghassemi and Aven Zitzelberger and Jessica Lundin and Guillaume Chabot-Couture},
year={2024}
}
Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments
1
3 commits
3 linked in READMEs
updated Dec 20, 2024
Authors: Tuka Alhanai [email protected], Adam Kasumovic [email protected], Mohammad Ghassemi [email protected], Aven Zitzelberger [email protected], Jessica Lundin [email protected], Guillaume Chabot-Couture [email protected]
This HuggingFace Dataset contains the human-translated benchmarks we created from our paper, titled as above. Find the paper here: https://arxiv.org/abs/2412.12417
For more information, see the full repository on GitHub: https://github.com/InstituteforDiseaseModeling/Bridging-the-Gap-Low-Resource-African-Languages
Be sure to run pip install datasets to install HuggingFace's datasets package first.
Adjust the top three variables as desired to specify the language, subject, and split of the dataset.
Compared to Winogrande, the MMLU subsets in this dataset have:
from datasets import load_dataset # pip install datasets
from pprint import pprint
from collections import Counter
# TODO: Developer set these three variables as desired
# Afrikaans (af), Amharic (am), Bambara (bm), Igbo (ig), Sepedi (nso), Shona (sn),
# Sesotho (st), Setswana (tn), Tsonga (ts), Xhosa (xh), Zulu (zu)
desired_lang = "af"
# clinical_knowledge, college_medicine, virology
desired_subject = "clinical_knowledge"
# dev, test, val
desired_split = "test"
# Load dataset
dataset_path = "Institute-Disease-Modeling/mmlu-winogrande-afr"
desired_subset = f"mmlu_{desired_subject}_{desired_lang}"
dataset = load_dataset(dataset_path, desired_subset, split=desired_split)
# Inspect Dataset
# General Information
print("\nDataset Features:")
pprint(dataset.features)
print("\nNumber of rows in the dataset:")
print(len(dataset))
# Inspect Questions and Options
# Convert dictionary of lists to list of dictionaries for easier iteration
dataset_list = [dict(zip(dataset[:].keys(), values)) for values in zip(*dataset[:].values())]
print("\nExample Questions and Options:")
for row in dataset_list[:3]: # Inspect the first 3 rows
print(f"Question: {row['Question']}")
print(f"Options: A) {row['OptionA']} | B) {row['OptionB']} | C) {row['OptionC']} | D) {row['OptionD']}")
print(f"Answer: {row['Answer']}")
print("-" * 50)
# Analyze Answer Distribution
answer_distribution = Counter(row['Answer'] for row in dataset)
print("\nAnswer Distribution:")
for answer, count in sorted(answer_distribution.items()):
print(f"Answer {answer}: {count} ({count / len(dataset) * 100:.2f}%)")
# Average Question Length
avg_question_length = sum(len(row['Question']) for row in dataset) / len(dataset)
print(f"\nAverage Question Length: {avg_question_length:.2f} characters")
Be sure to run pip install datasets to install HuggingFace's datasets package first.
Adjust the top two variables as desired to specify the language and split of the dataset.
Compared to MMLU, the Winogrande subsets in this dataset have:
from datasets import load_dataset # pip install datasets
from pprint import pprint
from collections import Counter
# TODO: Developer set these two variables as desired
# Afrikaans (af), Amharic (am), Bambara (bm), Igbo (ig), Sepedi (nso), Shona (sn),
# Sesotho (st), Setswana (tn), Tsonga (ts), Xhosa (xh), Zulu (zu)
desired_lang = "bm"
# dev, test, train_s
desired_split = "train_s"
# Load dataset
dataset_path = "Institute-Disease-Modeling/mmlu-winogrande-afr"
desired_subset = f"winogrande_{desired_lang}"
dataset = load_dataset(dataset_path, desired_subset, split=desired_split)
# Inspect Dataset
# General Information
print("\nDataset Features:")
pprint(dataset.features)
print("\nNumber of rows in the dataset:")
print(len(dataset))
# Inspect Sentences and Options
# Convert dictionary of lists to list of dictionaries for easier iteration
dataset_list = [dict(zip(dataset[:].keys(), values)) for values in zip(*dataset[:].values())]
print("\nExample Sentences and Options:")
for row in dataset_list[:3]: # Inspect the first 3 rows
print(f"Sentence: {row['Sentence']}")
print(f"Options: 1) {row['Option1']} | 2) {row['Option2']}")
print(f"Answer: {row['Answer']}")
print("-" * 50)
# Analyze Answer Distribution
answer_distribution = Counter(row['Answer'] for row in dataset)
print("\nAnswer Distribution:")
for answer, count in sorted(answer_distribution.items()):
print(f"Answer {answer}: {count} ({count / len(dataset) * 100:.2f}%)")
# Average Sentence Length
avg_sentence_length = sum(len(row['Sentence']) for row in dataset) / len(dataset)
print(f"\nAverage Sentence Length: {avg_sentence_length:.2f} characters")
As used in our own experiments, we have prepared fine-tunable versions of the datasets (in GPT format), which are present in the GitHub repository. These datasets can be used with OpenAI's Fine-Tuning API to fine-tune GPT models on our MMLU and Winogrande translations. Note that since MMLU does not have a train set, the entirety of MMLU college medicine is used for training (MMLU college medicine is naturally excluded from testing for fine-tuned models).
Moreover, see here for an example Jupyter Notebook from our GitHub repository that allows the user to fine-tune a number of models by selecting the desired fine-tuning datasets. The notebook then fine-tunes Unsloth's Llama 3 70B IT (the model can be swapped out with similar models) on each fine-tuning dataset and evaluates each fine-tuned model's performance on MMLU and Winogrande test sets (the same as in this HuggingFace Dataset, but formatted into JSONL). Note that using the aforementioned notebook requires a full clone of the GitHub repository and a powerful GPU like a NVIDIA A100 GPU.
For more details, see our paper.
The code in this repository was developed by IDM, the Bill & Melinda Gates Foundation, and Ghamut Corporation to further research in Large Language Models (LLMs) for low-resource African languages by allowing them to be evaluated on question-answering and commonsense reasoning tasks, like those commonly available in English. We’ve made it publicly available under the MIT License to provide others with a better understanding of our research and an opportunity to build upon it for their own work. We make no representations that the code works as intended or that we will provide support, address issues that are found, or accept pull requests. You are welcome to create your own fork and modify the code to suit your own modeling needs as contemplated under the MIT License.
This HuggingFace Dataset includes data derived from the following datasets, each subject to their respective licenses (copied from their respective GitHub repositories):
@article{hendryckstest2021,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}
@article{hendrycks2021ethics,
title={Aligning AI With Shared Human Values},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}
@article{sakaguchi2019winogrande,
title={WinoGrande: An Adversarial Winograd Schema Challenge at Scale},
author={Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin},
journal={arXiv preprint arXiv:1907.10641},
year={2019}
}
Please note that the licenses for the included datasets are separate from and may impose additional restrictions beyond the HuggingFace Dataset's main license.
If you find this HuggingFace Dataset useful, please consider citing it:
@article{,
title={Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments},
author={Tuka Alhanai and Adam Kasumovic and Mohammad Ghassemi and Aven Zitzelberger and Jessica Lundin and Guillaume Chabot-Couture},
year={2024}
}