OpenMM-Medical

Introduction

OpenMM-Medical is a comprehensive medical evaluation dataset, which is an integration of existing datasets. OpenMM-Medical spans multiple domains, including Magnetic Resonance Imaging (MRI), CT scans, X-rays, microscopy images, endoscopy, fundus imaging, and dermoscopy.

Usage

The following steps detail how to use Baichuan-Omni-1.5 with OpenMM-Medical for evaluation using VLMEvalKit:

---

1. Add baichuan.py in VLMEvalKit/vlmeval/vlm

Download baichuan.py (which defines the Baichuan model class) and add it in VLMEvalKit/vlmeval/vlm.

---

2. Modify VLMEvalKit/vlmeval/vlm/__init__.py

Add the following line:

from .baichuan import Baichuan

---

3. Modify VLMEvalKit/vlmeval/config.py

Import the Baichuan model:

from vlmeval.vlm import Baichuan

Add the Baichuan-omni model configuration:

'Baichuan-omni': partial(
    Baichuan, 
    sft=True, 
    model_path='/your/path/to/the/model/checkpoint'
)

---

4. Modify VLMEvalKit/vlmeval/dataset/image_mcq.py

Download image_mcq.py and add the following code to define the OpenMMMedical class. Ensure the image_folder points to your OpenMM-Medical dataset location:

class OpenMMMedical(ImageMCQDataset):

    @classmethod
    def supported_datasets(cls):
        return ['OpenMMMedical']

    def load_data(self, dataset='OpenMMMedical'):
        image_folder = "/your/path/to/OpenMM_Medical"
        def generate_tsv(pth):
            import csv
            from pathlib import Path
            tsv_file_path = os.path.join(LMUDataRoot(), f'{dataset}.tsv')
        ...

---

5. Update VLMEvalKit/vlmeval/dataset/__init__.py

Import OpenMMMedical:

from .image_mcq import (
    ImageMCQDataset, MMMUDataset, CustomMCQDataset, 
    MUIRDataset, GMAIMMBenchDataset, MMERealWorld, OpenMMMedical
)

IMAGE_DATASET = [
    ImageCaptionDataset, ImageYORNDataset, ImageMCQDataset, ImageVQADataset,
    MathVision, MMMUDataset, OCRBench, MathVista, LLaVABench, MMVet,
    MTVQADataset, TableVQABench, MMLongBench, VCRDataset, MMDUDataset,
    DUDE, SlideVQA, MUIRDataset, GMAIMMBenchDataset, MMERealWorld, OpenMMMedical
]

---

6. Update VLMEvalKit/vlmeval/dataset/image_base.py

Modify the img_root_map function:

def img_root_map(dataset):
    if 'OpenMMMedical' in dataset:
        return 'OpenMMMedical'
    if 'OCRVQA' in dataset:
        return 'OCRVQA'
    if 'COCO_VAL' == dataset:
        return 'COCO'
    if 'MMMU' in dataset:
        return 'MMMU'

---

7. Run the Evaluation

Execute the following command to start the evaluation:

python run.py --data OpenMMMedical --model Baichuan-omni --verbose

---

Notes:

• Ensure that all paths (e.g., /your/path/to/OpenMM_Medical) are correctly specified.
• Confirm that the Baichuan model checkpoint is accessible at the defined model_path.
• Validate the dependencies and configurations of VLMEvalKit to avoid runtime issues.

With this setup, you should be able to evaluate OpenMM-Medical using Baichuan-Omni successfully.