AI Medical Dataset

Introduction

The AI Medical General Dataset is an experimental dataset designed to build a general chatbot with a strong foundation in medical knowledge. This dataset provides a large corpus of medical data, consisting of approximately 27 million rows, specifically adapted for training Large Language Models (LLMs) in the medical domain.

Data Sources

Our dataset is comprised of three primary sources:

These sources provide a diverse range of medical information, ensuring that the chatbot can be trained to understand and respond to a wide range of medical queries and topics.

Dataset Statistics

• Total rows: 27,000,000
• Total words: approximately 1,107.8M

Dataset Structure

The dataset is stored in Parquet format and consists of two columns:

• question: the medical question
• context: the corresponding context related to the question

The dataset contains approximately 27 million questions, each with its corresponding context, providing a rich source of medical information for training and fine-tuning language models.

Installation

!pip install datasets

from datasets import load_dataset
dataset = load_dataset("ruslanmv/ai-medical-dataset")

# Select the first 10 rows of the dataset
first_10_rows = dataset["train"].select(range(10))
# Convert the first 10 rows to a pandas DataFrame
first_10_rows_df = first_10_rows.to_pandas()

License

The AI Medical General Dataset is licensed under the CC-BY 4.0 license.

Citation

u use this dataset in your research, please cite it as follows:

@dataset{ai_medical_dataset,
  title = {AI Medical Dataset},
  author = {Ruslan Magana Vsevolodovna},
  year = {2023},
  url = {https://github.com/ruslanmv/ai-medical-chatbot},
}

Acknowledgments

We would like to acknowledge the contributions of the original data sources: ClinicalTrials, EMEA, and PubMed.

Contact

For any questions, issues, or feedback, please feel free to open an issue on this repository or contact us at contact@ruslanmv.com. You can also visit our website at ruslannmv.com for more information.