Awesome Vision-Language Models (VLMs) for Medical Report Generation (RG) and Visual Question Answering (VQA)

Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review is the comprehensive review that includes:

• the latest publicly available VLMs specifically designed for medical RG and VQA;
• the essential background on computer vision, natural language processing, and VLMs to ensure its accessibility for readers without a machine learning background;
• the description of publicly available vision-language datasets, encompassing medical image-text pairs or question-answer pairs related to medical images;
• the detailed description of metrics employed for evaluating VLMs on RG and VQA tasks;
• the discussion of current challenges in the field and various potential research directions that could significantly shape the future of medical VLMs.

The list of medical VLMs

The list of Medical Vision-Language Datasets

Citation

@article{Hartsock2024,
  title={Vision-language models for medical report generation and visual question answering: a review},
  author={Hartsock, Iryna and Rasool, Ghulam},
  journal={Frontiers in Artificial Intelligence},
  volume={7},
  pages={1430984},
  year={2024},
  publisher={Frontiers Media SA},
  doi={10.3389/frai.2024.1430984},
  url={https://www.frontiersin.org/articles/10.3389/frai.2024.1430984/full}
}