CareQA

---

Table of Contents

• Table of Contents
• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Distribution: CareQA HuggingFace
• Paper: Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
• Point of Contact: Pablo Bernabeu

Dataset Summary

CareQA is a healthcare QA dataset with two versions:

• Closed-Ended Version: A multichoice question answering (MCQA) dataset containing 5,621 QA pairs across six categories. Available in English and Spanish.
• Open-Ended Version: A free-response dataset derived from the closed version, containing 2,769 QA pairs (English only).

The dataset originates from official sources of the Spanish Specialized Healthcare Training (FSE) examinations, including the biology, chemistry, medicine, nursing, pharmacology, and psychology tests from 2020 to 2024. The English translation was performed using GPT-4, and the open-ended version was created via rephrasing with Qwen2.5-72B-Instruct, followed by human validation. CareQA was originally built to evaluate models of the Aloe family. See how it was used in the correponding papers of the Alpha and Beta versions.

Supported Tasks and Leaderboards

• Multichoice QA (Closed-Ended): Models select the correct answer from four options. Evaluated via accuracy.
• Open-Ended QA: Models generate free-form answers. Evaluated using n-gram metrics, semantic similarity, perplexity or LLM-as-a-judge.

Languages

• Closed-Ended: Spanish (original) and English (GPT-4 translated).
• Open-Ended: English only.

Dataset Structure

Data Instances

CareQA_es

{
    "exam_id":16,
    "question":"Las células de Leydig:",
    "op1":"Sintetizan testosterona a partir de colesterol.",
    "op2":"Establecen la barrera hemato-testicular.",
    "op3":"Sintetizan estradiol a partir de testosterona.",
    "op4":"Son la diana principal de la FSH.",
    "cop":1,
    "year":2021,
    "category":"Biology",
    "unique_id":"e06d5b5a-2fda-4586-a7cc-c1a2f1d5d0c7"
}

CareQA_en

{
    "exam_id":16,
    "question":"The Leydig cells:",
    "op1":"They synthesize testosterone from cholesterol.",
    "op2":"They establish the blood-testis barrier.",
    "op3":"They synthesize estradiol from testosterone.",
    "op4":"They are the main target of FSH.",
    "cop":1,
    "year":2021,
    "category":"Biology",
    "unique_id":"e06d5b5a-2fda-4586-a7cc-c1a2f1d5d0c7"
}

CareQA_en_open

{
    "exam_id":16,
    "question":"What is the primary function of Leydig cells in the synthesis of testosterone?",
    "answer":"They synthesize testosterone from cholesterol.",
    "year":2021,
    "category":"Biology",
    "unique_id":"e06d5b5a-2fda-4586-a7cc-c1a2f1d5d0c7"
}

Data Fields

Closed-Ended Splits:

• exam_id: question number in the original exam
• question: the question text
• op1: option 1
• op2: option 2
• op3: option 3
• op4: option 4
• cop: correct option (1-4)
• year: year of the exam
• category: subject category of the exam
• unique_id: unique identifier for each question

Open-Ended Split:

• exam_id: question number in the original exam
• question: the rephrased question text
• answer: the correct option
• year: year of the exam
• category: subject category of the exam
• unique_id: unique identifier for each question

Data Splits

The dataset is conceived as an evaluation dataset, so the whole dataset is considered as the test set.

Closed-Ended Splits:

The close-ended splits contain 5621 samples in total corresponding to six exam categories and five years of exams.

The distribution of samples per category is as follows:

• Biology: 966 samples
• Chemistry: 944 samples
• Medicine: 857 samples
• Nursing: 923 samples
• Pharmacology: 969 samples
• Psychology: 962 samples

The distribution of samples per year is as follows:

• 2020: 1037 samples
• 2021: 1037 samples
• 2022: 1182 samples
• 2023: 1188 samples
• 2024: 1177 samples

Open-Ended Split:

The open-ended split contains 2769 samples in total corresponding to six exam categories and five years of exams.

The distribution of samples per category is as follows:

• Biology: 556 samples
• Chemistry: 514 samples
• Medicine: 373 samples
• Nursing: 393 samples
• Pharmacology: 483 samples
• Psychology: 450 samples

The distribution of samples per year is as follows:

• 2020: 528 samples
• 2021: 501 samples
• 2022: 583 samples
• 2023: 601 samples
• 2024: 556 samples

Dataset Creation

Curation Rationale

There is a lack of question answering scientific and health-related datasets, especially in Spanish. This dataset aims to fill this gap by providing a dataset that includes the last five years of the Spanish Specialised Healthcare Training (FSE) exams, which encompass biology, chemistry, medicine, nursing, pharmacology and psychology.

Source Data

Data Collection and Normalization

The exams were collected from the official government website. The questions were extracted from the PDF files using an automatic process and then manually reviewed to ensure the quality of the dataset. Questions that include images or diagrams in the question text or options were discarded, as well as a small number of questions that included incorrectly parsed characters. The dataset does not contain questions that were annulled after the challenge period.

Translation

The Spanish-to-English translation was performed using GPT-4 (gpt-4-0314), followed by a validation process:

• Validation Protocol: 360 randomly selected samples (95% confidence level with a 5% margin of error) were reviewed by 10 annotators, with each sample evaluated by at least three annotators.
• Results: 83.1% of translations received unanimous annotator approval, with 96.7% being approved by at least two reviewers.

Rephrasal

The open-ended subset was developed through the following steps:

1. Rephrasing Process: Closed-ended questions were transformed into free-response format using the Qwen2.5-72B-Instruct model.
2. Initial Filtering: The dataset was reduced from 5,621 to 3,730 questions by removing unanswerable or ambiguous formats.
3. Human Validation: The same 360-sample review protocol used for translation validation was applied. A 65.8% annotator agreement rate was achieved, with 85.8% being approved by at least two reviewers.
4. Refinement: To improve results according to human feedback, addressed ambiguous answers by reprompting the model to account for nuanced responses. Excluded 961 questions with multiple valid interpretations. The final dataset contains 2,769 QA pairs with a 73.6% full annotator agreement rate and 92.8% approval for two out of three reviewers.

Who are the source language producers?

The source language producers are experts in each of the exam categories.

Personal and Sensitive Information

The dataset does not contain any personal or sensitive information.

Considerations for Using the Data

Social Impact of Dataset

This dataset can be used to test models for question answering in the health domain. It is intended to be used to assess the medical knowledge of artificial intelligence models which may be used in clinical decision support systems, by people with difficulties accessing healthcare, or in other applications.

Discussion of Biases

The dataset has not been studied for bias assessment. However, as the original material has been produced by experts, we consider that the dataset is mostly free of biases. That said, it may not be representative of the full range of medical knowledge or all possible patient profiles (e.g., age, sex, ethnicity).

Other Known Limitations

The dataset is designed primarily for evaluation purposes and does not include a train/test split.

While the data has been carefully parsed from PDF files and manually reviewed, minor parsing errors may still exist. The translation from Spanish to English, performed using GPT-4, has undergone validation, but some translation errors may remain. Additionally, the dataset is limited to questions from the last five years of the Spanish Specialised Healthcare Training (FSE) exams, which may not fully represent the breadth of medical knowledge across all domains or time periods.

For the open-ended version, despite the refinement process and human validation, rephrasing the questions using Qwen2.5-72B-Instruct may introduce errors or subtle inaccuracies. The exclusion of ambiguous questions during the rephrasing process also means that the open-ended subset is smaller and may not cover the same scope as the closed-ended version.

If you encounter any issues with the dataset, please open an issue on the dataset repository.

Additional Information

Dataset Curators

Lucia Urcelay Ganzabal and Pablo Bernabeu Pérez.

Licensing Information

The dataset is licensed under the Apache License 2.0.

Citation Information

@inproceedings{arias-duart-etal-2025-automatic,
    title = "Automatic Evaluation of Healthcare {LLM}s Beyond Question-Answering",
    author = "Arias-Duart, Anna  and
      Martin-Torres, Pablo Agustin  and
      Hinjos, Daniel  and
      Bernabeu-Perez, Pablo  and
      Ganzabal, Lucia Urcelay  and
      Mallo, Marta Gonzalez  and
      Gururajan, Ashwin Kumar  and
      Lopez-Cuena, Enrique  and
      Alvarez-Napagao, Sergio  and
      Garcia-Gasulla, Dario",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-short.10/",
    pages = "108--130",
    ISBN = "979-8-89176-190-2",
    abstract = "Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model`s capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark{--}CareQA{--}, with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations {--}Relaxed Perplexity{--} to mitigate the identified limitations."
}