Maintainers

• Junfeng Jiang@Aizawa Lab: jiangjf (at) is.s.u-tokyo.ac.jp
• Jiahao Huang@Aizawa Lab: jiahao-huang (at) g.ecc.u-tokyo.ac.jp

If you find any error in this benchmark or want to contribute to this benchmark, please feel free to contact us.

Introduction

This is a dataset collection of JMedBench, which is a benchmark for evaluating Japanese biomedical large language models (LLMs). Details can be found in this paper. We also provide an evaluation framework, med-eval, for easy evaluation.

The JMedBench consists of 20 datasets across 5 tasks, listed below.

Limitations

Please be aware of the risks, biases, and limitations of this benchmark.

As introduced in the previous section, some evaluation datasets are translated from the original sources (in English). Although we used the most powerful API from OpenAI (i.e., gpt-4-0613) to conduct the translation, it may be unavoidable to contain incorrect or inappropriate translations. If you are developing biomedical LLMs for real-world applications, please conduct comprehensive human evaluation before deployment.

Citation

BibTeX: If our JMedBench is helpful for you, please cite our work:

@misc{jiang2024jmedbenchbenchmarkevaluatingjapanese,
      title={JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models}, 
      author={Junfeng Jiang and Jiahao Huang and Akiko Aizawa},
      year={2024},
      eprint={2409.13317},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.13317}, 
}