README
BrainFLORA Dataset and Checkpoints
This repository hosts the released assets for BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings.
BrainFLORA aligns EEG, MEG, and fMRI signals with visual-language representations for visual retrieval, image reconstruction, and image captioning.
Checkpoints
| File | Purpose |
|---|---|
checkpoints/eeg_01-06_01-46_150.pth | EEG single-modality retrieval encoder |
checkpoints/meg_01-11_14-50_150.pth | MEG single-modality retrieval encoder |
checkpoints/fmri_01-18_01-35_150.pth | fMRI single-modality retrieval encoder |
checkpoints/Unified_EEG+MEG+fMRI_EEG_01-27_02-32_60.pth | Unified EEG+MEG+fMRI retrieval model |
checkpoints/reconstruction_checkpoints/150.pth | Unified high-level reconstruction encoder |
checkpoints/reconstruction_checkpoints/prior_diffusion/150.pth | Reconstruction diffusion prior |
checkpoints/caption_checkpoints/90.pth | Caption-aligned unified encoder |
checkpoints/caption_checkpoints/prior_diffusion/100.pth | Caption diffusion prior |
External Caption Model
The Shikra model and mm_projector.bin are not hosted in this dataset repository. Download them from the original Shikra resources and place them under the code repository as:
external_models/shikra-7b
external_models/mm_projector.bin
References:
- Shikra paper: https://arxiv.org/abs/2306.15195
- Shikra code and model instructions: https://github.com/shikras/shikra
- Hugging Face model page: https://huggingface.co/shikras/shikra-7b-delta-v1
Usage
Clone or download this dataset repository, then follow the reproduction commands in the BrainFLORA code repository README.
git clone https://github.com/ncclab-sustech/BrainFLORA.git
cd BrainFLORA
Restore the downloaded assets so the code repository has the expected layout:
BrainFLORA/checkpoints/
BrainFLORA/features/
BrainFLORA/fmri_dataset/
BrainFLORA/meg_dataset/
Related Projects
BrainFLORA uses BrainHub-style caption metrics for caption evaluation. Please also cite the UMBRAE/BrainHub project when using these evaluation components.
- UMBRAE project: https://weihaox.github.io/UMBRAE/
- UMBRAE code: https://github.com/weihaox/UMBRAE
- BrainHub benchmark: https://github.com/weihaox/BrainHub
Citation
@inproceedings{li2025brainflora,
author = {Li, Dongyang and Qin, Haoyang and Wu, Mingyang and Wei, Chen and Liu, Quanying},
title = {BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings},
year = {2025},
isbn = {9798400720352},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746027.3754996},
doi = {10.1145/3746027.3754996},
booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia},
pages = {5577--5586}
}
@inproceedings{xia2024umbrae,
author = {Xia, Weihao and de Charette, Raoul and Oztireli, Cengiz and Xue, Jing-Hao},
title = {UMBRAE: Unified Multimodal Brain Decoding},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024}
}
Similar repos
Linked in READMEs
README
BrainFLORA Dataset and Checkpoints
This repository hosts the released assets for BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings.
BrainFLORA aligns EEG, MEG, and fMRI signals with visual-language representations for visual retrieval, image reconstruction, and image captioning.
Checkpoints
| File | Purpose |
|---|---|
checkpoints/eeg_01-06_01-46_150.pth | EEG single-modality retrieval encoder |
checkpoints/meg_01-11_14-50_150.pth | MEG single-modality retrieval encoder |
checkpoints/fmri_01-18_01-35_150.pth | fMRI single-modality retrieval encoder |
checkpoints/Unified_EEG+MEG+fMRI_EEG_01-27_02-32_60.pth | Unified EEG+MEG+fMRI retrieval model |
checkpoints/reconstruction_checkpoints/150.pth | Unified high-level reconstruction encoder |
checkpoints/reconstruction_checkpoints/prior_diffusion/150.pth | Reconstruction diffusion prior |
checkpoints/caption_checkpoints/90.pth | Caption-aligned unified encoder |
checkpoints/caption_checkpoints/prior_diffusion/100.pth | Caption diffusion prior |
External Caption Model
The Shikra model and mm_projector.bin are not hosted in this dataset repository. Download them from the original Shikra resources and place them under the code repository as:
external_models/shikra-7b
external_models/mm_projector.bin
References:
- Shikra paper: https://arxiv.org/abs/2306.15195
- Shikra code and model instructions: https://github.com/shikras/shikra
- Hugging Face model page: https://huggingface.co/shikras/shikra-7b-delta-v1
Usage
Clone or download this dataset repository, then follow the reproduction commands in the BrainFLORA code repository README.
git clone https://github.com/ncclab-sustech/BrainFLORA.git
cd BrainFLORA
Restore the downloaded assets so the code repository has the expected layout:
BrainFLORA/checkpoints/
BrainFLORA/features/
BrainFLORA/fmri_dataset/
BrainFLORA/meg_dataset/
Related Projects
BrainFLORA uses BrainHub-style caption metrics for caption evaluation. Please also cite the UMBRAE/BrainHub project when using these evaluation components.
- UMBRAE project: https://weihaox.github.io/UMBRAE/
- UMBRAE code: https://github.com/weihaox/UMBRAE
- BrainHub benchmark: https://github.com/weihaox/BrainHub
Citation
@inproceedings{li2025brainflora,
author = {Li, Dongyang and Qin, Haoyang and Wu, Mingyang and Wei, Chen and Liu, Quanying},
title = {BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings},
year = {2025},
isbn = {9798400720352},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746027.3754996},
doi = {10.1145/3746027.3754996},
booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia},
pages = {5577--5586}
}
@inproceedings{xia2024umbrae,
author = {Xia, Weihao and de Charette, Raoul and Oztireli, Cengiz and Xue, Jing-Hao},
title = {UMBRAE: Unified Multimodal Brain Decoding},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024}
}