MexExpQA: Multilingual Benchmarking of Medical QA with reference gold explanations and Retrieval Augmented Generation (RAG)

We present a new multilingual parallel medical benchmark, MedExpQA, for the evaluation of LLMs on Medical Question Answering. This benchmark can be used for various NLP tasks including: Medical Question Answering or Explanation Generation.

Although the design of MedExpQA is independent of any specific dataset, for the first version of the MedExpQA benchmark we leverage the commented MIR exams from the Antidote CasiMedicos dataset which includes gold reference explanations, which is currently available for 4 languages: English, French, Italian and Spanish.

• 📖 Paper:MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
• 💻 Github Repo (Data and Code): https://github.com/hitz-zentroa/MedExpQA
• 🌐 Project Website: https://univ-cotedazur.eu/antidote
• Funding: CHIST-ERA XAI 2019 call. Antidote (PCI2020-120717-2) funded by MCIN/AEI /10.13039/501100011033 and by European Union NextGenerationEU/PRTR

Example of Document in Antidote CasiMedicos Dataset

In this repository you can find the following data:

• casimedicos-raw: The textual content including Clinical Case (C), Question (Q), Possible Answers (P), and Explanation (E) as shown in the example above.
• casimedicos-exp: The manual annotations linking the explanations of the correct and incorrect possible answers.
• MedExpQA: benchmark for Medical QA based on gold reference explanations from casimedicos-exp and knowledge automatically extracted using RAG methods.

Data Explanation

The following attributes composed casimedicos-raw:

• id: unique doc identifier.
• year: year in which the exam was published by the Spanish Ministry of Health.
• question_id_specific: id given to the original exam published by the Spanish Ministry of Health.
• full_question: Clinical Case (C) and Question (Q) as illustrated in the example document above.
• full answer: Full commented explanation (E) as illustrated in the example document above.
• type: medical speciality.
• options: Possible Answers (P) as illustrated in the example document above.
• correct option: solution to the exam question.

Additionally, the following jsonl attribute was added to create casimedicos-exp:

• explanations: for each possible answer above, manual annotation states whether:
  1. the explanation for each possible answer exists in the full comment (E) and
  2. if present, then we provide character and token offsets plus the text corresponding to the explanation for each possible answer.

For MedExpQA benchmarking we have added the following elements in the data:

• rag
  1. clinical_case_options/MedCorp/RRF-2: 32 snippets extracted from the MedCorp corpus using the combination of clinical case and options as a query during the retrieval process. These 32 snippets are the resulting RRF combination of 32 separately retrieved snippets using BM25 and MedCPT.

MedExpQA Benchmark Overview

Prompt Example for LLMs

Benchmark Results (averaged per type of external knowledge for grounding)

LLMs evaluated: LLaMA, PMC-LLaMA, Mistral and BioMistral.

Citation

If you use MedExpQA then please cite the following paper:

@article{ALONSO2024102938,
title = {MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering},
journal = {Artificial Intelligence in Medicine},
pages = {102938},
year = {2024},
issn = {0933-3657},
doi = {https://doi.org/10.1016/j.artmed.2024.102938},
url = {https://www.sciencedirect.com/science/article/pii/S0933365724001805},
author = {Iñigo Alonso and Maite Oronoz and Rodrigo Agerri},
keywords = {Large Language Models, Medical Question Answering, Multilinguality, Retrieval Augmented Generation, Natural Language Processing},
}

Contact: Iñigo Alonso and Rodrigo Agerri HiTZ Center - Ixa, University of the Basque Country UPV/EHU