MMedC

💻Github Repo 🖨️arXiv Paper

The official pre-training dataset for "Towards Building Multilingual Language Model for Medicine".

News

• We add Arabic and German corpus to MMedC.

Introduction

This repo contains MMedC, a multilingual medical corpus with 25.5 billion tokens.

• English Textbooks is not included in this repo due to copyright issues. For this part of 4B English corpus, please refer to PMC-LLaMA

You can download the MMedC.zip file to access all the data. The data are saved in txt format, and the zip file contains four folders corresponding to four types of data sources: filtering content, medical websites, medical textbooks, and small-scale datasets. Please refer to our paper for details.

You can use the following method to obtain the paths to all txt files in the directory. Afterward, you can read these txt files and customize subsequent operations.

import os
txt_root = "PATH/TO/MMEDC"
txt_paths = []
for root, dirs, files in os.walk(txt_root):
    if 'cultural_filtered_data_used' not in root:
        for file in files:
            if file.endswith('.txt'):
                txt_paths.append(os.path.join(root, file))

Our GitHub provides a data collection pipeline as well as our data preprocessing code.

News

[2024.2.21] Our pre-print paper is released ArXiv. Dive into our findings here.

[2024.2.20] We release MMedLM and MMedLM 2. With an auto-regressive continues training on MMedC, these models achieves superior performance compared to all other open-source models, even rivaling GPT-4 on MMedBench.

[2023.2.20] We release MMedC, a multilingual medical corpus containing 25.5B tokens.

[2023.2.20] We release MMedBench, a new multilingual medical multi-choice question-answering benchmark with rationale. Check out the leaderboard here.

Evaluation on MMedBench

The further pretrained MMedLM 2 showcast it's great performance in medical domain across different language.

• GPT and Gemini is evluated under zero-shot setting through API
• Open-source models first undergo training on the trainset of MMedBench before evaluate.

Contact

If you have any question, please feel free to contact qiupengcheng@pjlab.org.cn.

Citation

@misc{qiu2024building,
      title={Towards Building Multilingual Language Model for Medicine}, 
      author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2024},
      eprint={2402.13963},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}