The Bones and Joints Benchmark

Dataset Overview

We have developed a specialized dataset focused on musculoskeletal disorders, designed to systematically evaluate the clinical capabilities of visual language models (VLMs). The evaluation covers knowledge recall, clinical note interpretation, radiology image interpretation, diagnosis generation and rationale, treatment planning and rationale. The dataset primarily includes multiple-choice questions and open-ended questions to comprehensively cover key evaluation dimensions.

Files

This repository contains two key files:

1. Questions: This file contains all types of questions, including multiple-choice questions and open-ended questions.

2. Ground Truth: This file provides ground truth answers associated with all the questions in the dataset.

Usage

To use this dataset for your research or experiment:

1. Download: Obtain the dataset file from this repository.

2. Loading: Import the dataset into your data analysis or deep learning tool.

3. Evaluation: Assess the accuracy of the output answers.

Evaluation Metrics

For MCQs in the dataset, we adopt Accuracy as the evaluation metric.

Accuracy= (Number of Correct Answers)/(Total Number of Questions)×100%

For open-ended questions, we have designed a dual evaluation process that combines automatic and human evaluation.

1. Automatic evaluation: Relying on LLMs for semantic similarity assessment.

2. Human evaluation: Senior experts assess the correctness of the model output.