Chest X-ray Image Captioning

Project Description

This research project, conducted as part of a university module, aims to design a base model for automated chest X-ray report generation using natural image-captioning techniques. The project leverages the success of Large Language Models (LLMs) and their integration into the vision domain (VLLM).

The models are contained in 'src/cxr_to_report/'. The other folder contain code for preprocessing the reports, such as removing NaN's from the CXR-Pro dataset and applying the recommanded split, or bringing the reports in a from the dataloader readable json-format. The third folder contains fine-tuning steps for the decoder.

Three models are were build, two utilizing the classic transformer architecture with corss-attention, and a third model, where the encoder-decoder communication is limited to the mapper-layer. The two first basemodels differ in decoders (GPT2-small versus GPT2-medium).