This HF dataset contains the information of the StatPearls corpus used in MedRAG. It can be used for medical Retrieval-Augmented Generation (RAG).
According to the privacy policy of StatPearls, we are not allowed to distribute the StatPearls content. Please download the raw data from NCBI Bookshelf and use our code for the chunking.
StatPearls is a point-of-the-care clinical decision support tool similar to UpToDate. We use the 9,330 publicly available StatPearl articles through NCBI Bookshelf to construct the StatPearls corpus. We chunked StatPearls according to the hierarchical structure, treating each paragraph in an article as a snippet and splicing all the relevant hierarchical headings as the corresponding title. Our chunked corpus contains 301,202 snippets with an average of 119 tokens.
Each row is a snippet of StatPearls, which includes the following features:
git clone https://github.com/Teddy-XiongGZ/MedRAG.git
cd MedRAG
wget https://ftp.ncbi.nlm.nih.gov/pub/litarch/3d/12/statpearls_NBK430685.tar.gz -P ./corpus/statpearls
tar -xzvf ./corpus/statpearls/statpearls_NBK430685.tar.gz -C ./corpus/statpearls
python src/data/statpearls.py
>> from src.medrag import MedRAG
>> question = "A lesion causing compression of the facial nerve at the stylomastoid foramen will cause ipsilateral"
>> options = {
"A": "paralysis of the facial muscles.",
"B": "paralysis of the facial muscles and loss of taste.",
"C": "paralysis of the facial muscles, loss of taste and lacrimation.",
"D": "paralysis of the facial muscles, loss of taste, lacrimation and decreased salivation."
}
>> medrag = MedRAG(llm_name="OpenAI/gpt-3.5-turbo-16k", rag=True, retriever_name="MedCPT", corpus_name="StatPearls")
>> answer, snippets, scores = medrag.answer(question=question, options=options, k=32) # scores are given by the retrieval system
@article{xiong2024benchmarking,
title={Benchmarking Retrieval-Augmented Generation for Medicine},
author={Guangzhi Xiong and Qiao Jin and Zhiyong Lu and Aidong Zhang},
journal={arXiv preprint arXiv:2402.13178},
year={2024}
}
This HF dataset contains the information of the StatPearls corpus used in MedRAG. It can be used for medical Retrieval-Augmented Generation (RAG).
According to the privacy policy of StatPearls, we are not allowed to distribute the StatPearls content. Please download the raw data from NCBI Bookshelf and use our code for the chunking.
StatPearls is a point-of-the-care clinical decision support tool similar to UpToDate. We use the 9,330 publicly available StatPearl articles through NCBI Bookshelf to construct the StatPearls corpus. We chunked StatPearls according to the hierarchical structure, treating each paragraph in an article as a snippet and splicing all the relevant hierarchical headings as the corresponding title. Our chunked corpus contains 301,202 snippets with an average of 119 tokens.
Each row is a snippet of StatPearls, which includes the following features:
git clone https://github.com/Teddy-XiongGZ/MedRAG.git
cd MedRAG
wget https://ftp.ncbi.nlm.nih.gov/pub/litarch/3d/12/statpearls_NBK430685.tar.gz -P ./corpus/statpearls
tar -xzvf ./corpus/statpearls/statpearls_NBK430685.tar.gz -C ./corpus/statpearls
python src/data/statpearls.py
>> from src.medrag import MedRAG
>> question = "A lesion causing compression of the facial nerve at the stylomastoid foramen will cause ipsilateral"
>> options = {
"A": "paralysis of the facial muscles.",
"B": "paralysis of the facial muscles and loss of taste.",
"C": "paralysis of the facial muscles, loss of taste and lacrimation.",
"D": "paralysis of the facial muscles, loss of taste, lacrimation and decreased salivation."
}
>> medrag = MedRAG(llm_name="OpenAI/gpt-3.5-turbo-16k", rag=True, retriever_name="MedCPT", corpus_name="StatPearls")
>> answer, snippets, scores = medrag.answer(question=question, options=options, k=32) # scores are given by the retrieval system
@article{xiong2024benchmarking,
title={Benchmarking Retrieval-Augmented Generation for Medicine},
author={Guangzhi Xiong and Qiao Jin and Zhiyong Lu and Aidong Zhang},
journal={arXiv preprint arXiv:2402.13178},
year={2024}
}