OmniMedVQA

We introduce OmniMedVQA, large-scale and comprehensive Visual Question Answering benchmark tailored to the medical domain. This benchmark is collected from 73 different medical datasets, contains 118,010 images with 127,995 QA-items, covering 12 different medical image modalities and referring to more than 20 human anatomical regions. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs.

Please visit the GitHub page and further exploit the dataset!

OmniMedVQA is built using multiple publicly available datasets. Due to data privacy and ethical requirements, we currently only offer access to a subset of OmniMedVQA. For open-access datasets, we provide the image and the corresponding QA information constructed on these datasets. For restricted-access datasets, we provide the relative paths to the image data in the original dataset, along with the QA information we have constructed. Please adhere to the respective dataset's license to access and use this data. We will continue to update and maintain this database. Please stay tuned for further updates from us.

👉 Filesystem Hierarchy

~/OmniMedVQA
├── Images
|      ├── ACRIMA
|      |      ├── Im002_ACRIMA.png
|      |      └──  ...
|      ├── Adam Challenge
|      └──  ...
├── QA_information
|      ├── Open-access
|      |     ├── ACRIMA.json
|      |     ├── Adam Challenge.json
|      |     └── ....
|      └── Restricted-access
|            ├── AIDA.json
|            ├── AIROGS.json
|            └── ...
|
└──README.md

In the Images directory, we only provide the images from the open-access dataset. In the QA_information directory, we provide QA information for both open-access and restricted-access data in the form of JSON files. The QA information includes image paths, corresponding image modality types, questions, question types, options, ground truth answers, and corresponding dataset name. The content is highly comprehensive. Here is an example of the information provided:

    {
        "dataset": "Covid CT",
        "question_id": "Covid CT_0082",
        "question_type": "Anatomy Identification",
        "question": "What anatomical area is shown in this picture?",
        "gt_answer": "Chest region.",
        "image_path": "Images/Covid CT/CT_COVID/bmj.m606.full-p4-22%3.png",
        "option_A": "Upper arm region",
        "option_B": "Chest region.",
        "option_C": "Leg region",
        "option_D": "Shoulder and upper back region",
        "modality_type": "CT(Computed Tomography)"
    },

The OmniMedVQA dataset contains a portion of medical images that have been split from 3D data. The naming convention for these sliced images follows the following format:

{dataset name}/{ori name}_{dimension_slice}.png

Note: "dataset name" refers to the specific dataset name that the case is from. "ori name" is the original case name in its dataset. "dimension slice", e.g., "x_100", indicates the dimension along which we split a 3D case as well as the slice ID in this dimension. If we split a 3D case with axis x and the current slice is 100, then the term can be "x_0100".

🤝 免责声明

• OmniMedVQA是基于多个公开的数据集构建，旨在取之于社区，回馈于社区，为研究人员和开发者提供一个用于学术和技术研究的资源。使用本数据集的任何个人或组织（以下统称为“使用者”）需遵守以下免责声明：

1. 数据集来源：本数据集基于多个公开的数据集构建，这些数据集的来源已在论文中明确标明。使用者应当遵守原始数据集的相关许可和使用条款。
2. 数据准确性：尽管我们已经努力确保数据集的准确性和完整性，但使用者应自行承担使用数据集可能带来的风险和责任。
3. 责任限制：在任何情况下，数据集的提供者及相关贡献者均不对使用者的任何行为或结果承担责任。
4. 使用约束：使用者在使用本数据集时，应遵守适用的法律法规和伦理规范。使用者不得将本数据集用于非法、侵犯隐私、诽谤、歧视或其他违法或不道德的目的。
5. 知识产权：本数据集所有影像数据的知识产权归原始数据集的相关权利人所有，使用者不得以任何方式侵犯数据集的知识产权。

• 作为非盈利机构，团队倡导和谐友好的开源交流环境，若在开源数据集内发现有侵犯您合法权益的内容，请联系我们，我们将尽最大努力协助您处理。

• 通过下载、复制、访问或使用本数据集，即表示使用者已阅读、理解并同意遵守本免责声明中的所有条款和条件。如果使用者无法接受本免责声明的任何部分，请勿使用本数据集。

🤝 Disclaimer

• OmniMedVQA is constructed based on multiple publicly available datasets and aims to provide a resource for academic and technical research to researchers and developers. Any individual or organization (hereinafter referred to as "User") using this dataset must comply with the following disclaimer:

1. Dataset Source: OmniMedVQA is constructed based on multiple publicly available datasets, and the sources of these datasets have been clearly indicated in the paper. Users should adhere to the relevant licenses and terms of use of the original datasets.
2. Data Accuracy: While efforts have been made to ensure the accuracy and completeness of the dataset, users assume all risks and liabilities associated with the use of the dataset.
3. Limitation of Liability: Under no circumstances shall the dataset providers or contributors be held liable for any actions or outcomes of the Users.
4. Usage Constraints: Users must comply with applicable laws, regulations, and ethical norms when using this dataset. The dataset must not be used for illegal, privacy-infringing, defamatory, discriminatory, or other unlawful or unethical purposes.
5. Intellectual Property: The intellectual property rights of the image data in this dataset belong to the relevant rights holders of the original datasets. Users must not infringe upon the intellectual property rights of the dataset in any way.

• As a non-profit organization, we advocate for a harmonious and friendly open-source communication environment. If any content in the open dataset is found to infringe upon your legitimate rights and interests, please contact us and we will make our best effort to assist you in addressing the issue.

• By downloading, copying, accessing, or using this dataset, the User indicates that they have read, understood, and agreed to comply with all the terms and conditions of this disclaimer. If the User cannot accept any part of this disclaimer, please refrain from using this dataset.

🤝 Acknowledgement

• We thank all medical workers and dataset owners for making public datasets available to the community. If you find that your dataset is included in our OmniMedVQA but you do not want us to do so, please contact us to remove it.

Reference

@article{hu2024omnimedvqa,
  title={OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM},
  author={Hu, Yutao and Li, Tianbin and Lu, Quanfeng and Shao, Wenqi and He, Junjun and Qiao, Yu and Luo, Ping},
  journal={arXiv preprint arXiv:2402.09181},
  year={2024}
}