Version 1.0 of the PARROT dataset, including more than 2600 human-generated radiology reports in 21 languages, free to use for researchers.
14
5 commits
updated May 3, 2025
Polyglot Annotated Radiological Reports for Open Testing
PARROT is an collaborative initiative to create a multilingual open dataset of radiological reports on which to test LLMs. The aim of PARROT is to represent the diversity of languages and reporting styles to promote applicability of LLM-related research in non-English clinical settings. PARROT is not-for-profit and does not recieve any funding at the moment.
Large language models (LLMs) have shown promising capabilities for structuring, simplifying and correcting radiological reports (1–3). However, the lack of open datasets in languages other than English restricts research to using data that cannot be shared to comply with privacy policies (4). This shortcoming makes the external validation of those emerging technologies challenging, hindering their diffusion and application in non-English speaking countries (5). A multilingual, open archive of radiological reports annotated by experts could circumvent this issue, by allowing researchers to test models on clinically pertinent tasks in various languages. Here, we present the Polyglot Annotated Radiological Reports for Open Testing (PARROT) project, intended to enhance the representation of the diversity of radiological reports across centers and languages, offering researchers a different perspective for testing LLMs on medical data.
PARROT is intended as a collaborative dataset:
Facilitating the access to realistic medical texts for the community
Addressing the representation of non-English medical data for testing LLMs
Assessing the ability of LLMs to generalize across diverse languages in clinical tasks
Limitations of the MIMIC dataset: The exclusively English datasets on which LLMs are currently tested in radiology, such as the mono-centric MIMIC, do not capture the diversity of radiological reports across centers and languages, and thus may hinder the applicability of those results in non-English speaking countries (6). Models are already trained on the MIMIC database, raising the issue of training and testing the LLMs ability on the same dataset (7). Lack of multilingual alternatives: Because of the private nature of radiological reports, there exists currently no available dataset of radiological reports beyond English. An alternative could be to translate the reports from the MIMIC database in other language, but they would fail to convey the true diversity of pathologies, examinations and reporting styles across countries. Rising need for diversity in benchmarks: The development of open-source lightweight models alleviates the financial barrier to accessing this technology, but makes the importance of diverse corpora on which to test them even more dire (8). Need for benchmarking LLMs on clinically pertinent tasks: Datasets are often crawled from websites and left un-annotated, hindering the complexity and diversity of tasks on which to test models. As a consequence, LLMs are currently benchmarked on tasks that do not represent their actual usefulness in clinical settings (7).
PARROT is an annotated open dataset of fictional reports written by radiologists in their native language. To circumvent privacy issues, the reports are completely fictional and as such can be shared without limitation. Because they are written by experts in their native language, they are expected to capture authentically the diversity of reporting styles and cultural specificity. The reports are annotated for five different tasks (translation, structuration, measurement extraction, correction of errors, ICD-10 code assignment), and centralized in an open archive. PARROT is intended as a test dataset for LLMs, and should not be used for training.
Radiologists from diverse languages, backgrounds and seniority can create fictional reports for exam modalities of their choice. The reports should be written as .txt, .rtf or .docx files. The authors must to translate the reports to English and complete the .csv with annotations. No background in informatics, AI or LLM is needed. Students are encouraged to participate if they had some experience with radiology, e.g. through an internship.
Engagement: reaching out to a variety of radiologists will be a critical challenge for PARROT in order to achieve a critical mass of reports in various languages. As such, participants are welcomed to invite colleagues to join the initiative. Scalability and maintenance: if PARROT is a success, the amount of time required to curate the database will become critical. As such, for step 2 of the project, financial support from institutions could become important, and tasks such as verifying reports, annotating them, and creating new tasks, could be crowd-sourced. At this stage of the project, we will actively seek for funding opportunities by national or international institutions (e.g., European Union, national research foundations) or industry partners. Broadening the scope: multilingual medical databases are rare and radiology reports do not encompass the entirety of contexts in which a LLM can be assessed in medicine. As such, physicians from other specialties will be welcomed to join the initiative in step 2 of the project.
As of v1.0 (may 2025), PARROT consists of:
PARROT v1.0 is released as a single .jsonl file (5.9MB), with the following structure:
data/
└── PARROT_v1_0.jsonl # one JSON per line
├─ no : int # report number
├─ language : str # report language
├─ modality : str # exam modality (CT, MRI, X-ray, etc)
├─ area : str # anatomical region (head, abdomen, chest, etc)
├─ report : str # original report
├─ translation : str # English translation
├─ report : str # original report
├─ icd : str # ICD-10 codes associated with the report
├─ contributor_code : str # name of the author
├─ country : str # country of origin
└─ subspecialty : str # two-letter code of the corresponding radiology subspecialties (NR: neuroradiology, GU: genito-urinary, etc)
PARROT is intended for testing, sharing ideas on use-cases of LLMs in radiology and medicine, and external validation of NLP approaches. Some use-cases are suggested in the examples section:
examples/
└── similar_reports.py # for finding the most similar reports based on their embeddings with free-text queries
└── reports_vizualisation.py # for creating an interactive maps of reports based on their embeddings
└── simplify_reports_ollama.py # for simplifying reports with either a priavacy preserving model...
└── simplify_reports_openai.py # ...or a cloud-based model
We would be pleased to include your reports in the next iterations of PARROT. Please contact [email protected] or create a pull request to move forward.
If you accept the data sharing agreement you will recieve by email, the reports will be shared publicly as part of PARROT database. Your reports will remain free for everyone to use, under CC BY-NC-SA 4.0 license.
As soon as you submit 20 reports with valid translation and ICD codes, or 40 reports with translations without ICD codes, or 40 ICD codes for reports without codes, or any association, you will be invited to join the PARROT Consortium of Authors of the dataset.
Great! PARROT is open-source, so you can start to annotate the dataset yourself, and engage other people to do so, either on the Discord server or via social media or by contacting [email protected]
PARROT is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) https://creativecommons.org/licenses/by-nc-sa/4.0/ This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
PARROT is a not-for-profit project imagined by Bastien Le Guellec (Lille, France) and Keno Bressem (Munich, Germany), with the support of Grégory Kuchcinki (Lille, France). At the moment, it recieves no institutional funding. If your are an institution wishing to contribute to the project, please send an email to [email protected]
Version 1.0 of the PARROT dataset, including more than 2600 human-generated radiology reports in 21 languages, free to use for researchers.
14
5 commits
updated May 3, 2025
Polyglot Annotated Radiological Reports for Open Testing
PARROT is an collaborative initiative to create a multilingual open dataset of radiological reports on which to test LLMs. The aim of PARROT is to represent the diversity of languages and reporting styles to promote applicability of LLM-related research in non-English clinical settings. PARROT is not-for-profit and does not recieve any funding at the moment.
Large language models (LLMs) have shown promising capabilities for structuring, simplifying and correcting radiological reports (1–3). However, the lack of open datasets in languages other than English restricts research to using data that cannot be shared to comply with privacy policies (4). This shortcoming makes the external validation of those emerging technologies challenging, hindering their diffusion and application in non-English speaking countries (5). A multilingual, open archive of radiological reports annotated by experts could circumvent this issue, by allowing researchers to test models on clinically pertinent tasks in various languages. Here, we present the Polyglot Annotated Radiological Reports for Open Testing (PARROT) project, intended to enhance the representation of the diversity of radiological reports across centers and languages, offering researchers a different perspective for testing LLMs on medical data.
PARROT is intended as a collaborative dataset:
Facilitating the access to realistic medical texts for the community
Addressing the representation of non-English medical data for testing LLMs
Assessing the ability of LLMs to generalize across diverse languages in clinical tasks
Limitations of the MIMIC dataset: The exclusively English datasets on which LLMs are currently tested in radiology, such as the mono-centric MIMIC, do not capture the diversity of radiological reports across centers and languages, and thus may hinder the applicability of those results in non-English speaking countries (6). Models are already trained on the MIMIC database, raising the issue of training and testing the LLMs ability on the same dataset (7). Lack of multilingual alternatives: Because of the private nature of radiological reports, there exists currently no available dataset of radiological reports beyond English. An alternative could be to translate the reports from the MIMIC database in other language, but they would fail to convey the true diversity of pathologies, examinations and reporting styles across countries. Rising need for diversity in benchmarks: The development of open-source lightweight models alleviates the financial barrier to accessing this technology, but makes the importance of diverse corpora on which to test them even more dire (8). Need for benchmarking LLMs on clinically pertinent tasks: Datasets are often crawled from websites and left un-annotated, hindering the complexity and diversity of tasks on which to test models. As a consequence, LLMs are currently benchmarked on tasks that do not represent their actual usefulness in clinical settings (7).
PARROT is an annotated open dataset of fictional reports written by radiologists in their native language. To circumvent privacy issues, the reports are completely fictional and as such can be shared without limitation. Because they are written by experts in their native language, they are expected to capture authentically the diversity of reporting styles and cultural specificity. The reports are annotated for five different tasks (translation, structuration, measurement extraction, correction of errors, ICD-10 code assignment), and centralized in an open archive. PARROT is intended as a test dataset for LLMs, and should not be used for training.
Radiologists from diverse languages, backgrounds and seniority can create fictional reports for exam modalities of their choice. The reports should be written as .txt, .rtf or .docx files. The authors must to translate the reports to English and complete the .csv with annotations. No background in informatics, AI or LLM is needed. Students are encouraged to participate if they had some experience with radiology, e.g. through an internship.
Engagement: reaching out to a variety of radiologists will be a critical challenge for PARROT in order to achieve a critical mass of reports in various languages. As such, participants are welcomed to invite colleagues to join the initiative. Scalability and maintenance: if PARROT is a success, the amount of time required to curate the database will become critical. As such, for step 2 of the project, financial support from institutions could become important, and tasks such as verifying reports, annotating them, and creating new tasks, could be crowd-sourced. At this stage of the project, we will actively seek for funding opportunities by national or international institutions (e.g., European Union, national research foundations) or industry partners. Broadening the scope: multilingual medical databases are rare and radiology reports do not encompass the entirety of contexts in which a LLM can be assessed in medicine. As such, physicians from other specialties will be welcomed to join the initiative in step 2 of the project.
As of v1.0 (may 2025), PARROT consists of:
PARROT v1.0 is released as a single .jsonl file (5.9MB), with the following structure:
data/
└── PARROT_v1_0.jsonl # one JSON per line
├─ no : int # report number
├─ language : str # report language
├─ modality : str # exam modality (CT, MRI, X-ray, etc)
├─ area : str # anatomical region (head, abdomen, chest, etc)
├─ report : str # original report
├─ translation : str # English translation
├─ report : str # original report
├─ icd : str # ICD-10 codes associated with the report
├─ contributor_code : str # name of the author
├─ country : str # country of origin
└─ subspecialty : str # two-letter code of the corresponding radiology subspecialties (NR: neuroradiology, GU: genito-urinary, etc)
PARROT is intended for testing, sharing ideas on use-cases of LLMs in radiology and medicine, and external validation of NLP approaches. Some use-cases are suggested in the examples section:
examples/
└── similar_reports.py # for finding the most similar reports based on their embeddings with free-text queries
└── reports_vizualisation.py # for creating an interactive maps of reports based on their embeddings
└── simplify_reports_ollama.py # for simplifying reports with either a priavacy preserving model...
└── simplify_reports_openai.py # ...or a cloud-based model
We would be pleased to include your reports in the next iterations of PARROT. Please contact [email protected] or create a pull request to move forward.
If you accept the data sharing agreement you will recieve by email, the reports will be shared publicly as part of PARROT database. Your reports will remain free for everyone to use, under CC BY-NC-SA 4.0 license.
As soon as you submit 20 reports with valid translation and ICD codes, or 40 reports with translations without ICD codes, or 40 ICD codes for reports without codes, or any association, you will be invited to join the PARROT Consortium of Authors of the dataset.
Great! PARROT is open-source, so you can start to annotate the dataset yourself, and engage other people to do so, either on the Discord server or via social media or by contacting [email protected]
PARROT is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) https://creativecommons.org/licenses/by-nc-sa/4.0/ This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
PARROT is a not-for-profit project imagined by Bastien Le Guellec (Lille, France) and Keno Bressem (Munich, Germany), with the support of Grégory Kuchcinki (Lille, France). At the moment, it recieves no institutional funding. If your are an institution wishing to contribute to the project, please send an email to [email protected]