Accepted at EMNLP 2025 Findings 🏅

BiMediX2 : Bio-Medical EXpert LMM for Diverse Medical Modalities

Sahal Shaji Mullappilly*, Mohammed Irfan K*, Sara Pieri, Saeed Yahya Alseiari, Shanavas Cholakkal, Khaled Aldahmani, Fahad Khan, Rao Muhammad Anwer, Salman Khan, Timothy Baldwin, and Hisham Cholakkal

*Equally contributing first authors

Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), UAE

📢 Latest Updates

• Oct-20-25: Our Training configurations, Data, and evaluation framework are released! 🔥
• Dec-15-24: Our Model checkpoints are released on HuggingFace link. 🔥
• Dec-11-24: BiMediX2 Technical Report is released link. 🔥
• Dec-02-24: BiMediX2 is featured by AI at Meta link. 🔥
• Sep-24-24: BiMediX2 wins the Meta Llama Impact Innovation Award link. 🔥

👩‍⚕️ Overview

Introducing BiMediX2, the first bilingual (Arabic-English) Bio-Medical Expert Large Multimodal Model (LMM) designed for advanced medical image understanding and applications. Built on the Llama 3.1 architecture, BiMediX2 seamlessly integrates text and visual modalities to enable multilingual interactions, including text-based queries and multi-turn conversations involving medical images. Trained on a diverse bilingual and multimodal healthcare dataset of 1.6M samples, it achieves state-of-the-art performance across various benchmarks. BiMediX2 outperforms recent models in multimodal medical evaluations, delivering over 9% improvement in English and 20% in Arabic evaluations, and excelling in tasks like medical VQA, Report Generation, and Summarization.

---

🏆 Contributions

Our key contributions are as follows:

• We introduce the first bilingual medical LMM that achieves state-of-the-art results on VLM evaluation benchmarks across various medical image modalities, while also excelling on medical LLM evaluation benchmarks.
• We curated a comprehensive Arabic-English multimodal bilingual instruction set named BiMed-V comprising over 1.6M instructions.
• We introduce the first bilingual GPT-4o-based medical LMM benchmark named BiMed-MBench, consisting of 386 medical queries in English and Arabic across various medical image modalities, fully verified by medical experts.
• Our BiMediX2 LLM outperforms GPT-4 by more than 8% on the USMLE benchmark and by more than 9% in UPHILL factual accuracy evaluations.
• Our BiMediX2 LMM achieves state-of-the-art results on BiMed-MBench, with over a 9% improvement in English evaluations and more than a 20% improvement in Arabic evaluations. Furthermore, it excels in medical Visual Question Answering, Report Generation, and Report Summarization tasks.

---

📊 Model Performance

BiMed-MBench Evaluation

Clinical LLM Evaluation

---

Training

For a quick start, here’s the minimal flow to train BiMediX2. See README.md for detailed steps and notes.

• Prerequisite: Install LLaVA-pp for Llama-3-V per their guide.
• Data: Download BiMed-V_stage1.json and BiMed-V_stage2.json (links in README.md). Download images using the provided scripts.
  • Images: PMC, Slake-VQA, Rad-VQA, Path-VQA. Use:
    • python scripts/download_pmc_images.py
    • python scripts/download_slake_rad_path_images.py
  • Update image folder path in the training scripts.
• Stage 1 (pretrain projector):
  • Run: ./scripts/stage1_train.sh
  • Output: ./checkpoints/BiMediX2_llava_8B_pretrain/ (contains mm_projector.bin)
• Stage 2 (LoRA finetune):
  • Run: ./scripts/stage2_train.sh
  • Loads projector from Stage 1; outputs LoRA to ./checkpoints/BiMediX2_llava_8B_finetune_lora/
• Optional: Merge LoRA with base model using ./scripts/merge_lora.sh.

BiMed-MBench Evaluation

To reproduce our BiMed-MBench evaluation results or evaluate your own models, please refer to the BiMed-MBench folder. This contains:

• Bilingual (English & Arabic) evaluation datasets
• Evaluation scripts for GPT-4o-based assessment
• Instructions for running evaluations on custom models

For detailed usage instructions, see the BiMed-MBench README.

---

֎ BiMediX2 Architecture

---

🌟 Examples

---

📜 License & Citation

BiMediX2 is released under the CC-BY-NC-SA 4.0 License. For more details, please refer to the LICENSE file included in our BiMediX repository.

⚠️ Warning! This release, intended for research, is not ready for clinical or commercial use.

Users are urged to employ BiMediX2 responsibly, especially when applying its outputs in real-world medical scenarios. It is imperative to verify the model's advice with qualified healthcare professionals and not rely on it for medical diagnoses or treatment decisions. Despite the overall advancements BiMediX2 shares common challenges with other language models, including hallucinations, toxicity, and stereotypes. BiMediX2's medical diagnoses and recommendations are not infallible.

If you use BiMediX2 in your research, please cite our work as follows:

@misc{mullappilly2024bimedix2biomedicalexpertlmm,
      title={BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities}, 
      author={Sahal Shaji Mullappilly and Mohammed Irfan Kurpath and Sara Pieri and Saeed Yahya Alseiari and Shanavas Cholakkal and Khaled Aldahmani and Fahad Khan and Rao Anwer and Salman Khan and Timothy Baldwin and Hisham Cholakkal},
      year={2024},
      eprint={2412.07769},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.07769}, 
}

---

🙏 Acknowledgements

We are thankful to Meta Llama for releasing their models and LLaVa, Axolotl, LLaVA++ for their open-source code contributions.

We would like to thank Dr. Omair Mohammed, Dr. Mohammed Zidan and Dr. Vishal Thomas Oommen for their contribution in verification of medical responses.

The computations were enabled by resources provided by LUMI hosted by CSC (Finland) and LUMI consortium, and by Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the NSC.

We are grateful to the Meta Llama Impact Innovation Awards for recognizing BiMediX2, as one of the winners awarded in September 2024. This recognition highlights our commitment to advancing AI-driven healthcare solutions.

---