FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens

Christian Schlarmann  •  Francesco Croce  •  Nicolas Flammarion  •  Matthias Hein

[Paper] [HuggingFace] [BibTeX]

FuseLIP is a multimodal embedding architecture that unifies text and image inputs through early fusion. Unlike traditional contrastive models that use separate encoders and rely on late fusion, FuseLIP employs a single transformer operating on a joint vocabulary of discrete image and text tokens. This enables deep cross-modal interaction and richer representations.

Table of Contents

• Installation
  • Dataset Preparation
• Pretrained Models
• Training
• Evaluation
• Generating VQA Data
• Citation

Installation

To install the required packages, install Python 3.11 and run:

conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.4 -c pytorch -c nvidia  # slightly different results when installing pytorch via pip
pip install -r requirements.txt
export PYTHONPATH="$PYTHONPATH:./src"

Dataset Preparation

For training FuseLIP, please gather the following datasets:

• CC3M and CC12M: we download the datasets from HuggingFace and extract the images and captions. Cleaned CSV files are available here (obtained via scripts/cc_wds_to_csv.py).
• CC3M-VQA
  • we also supply CC12M-VQA, but it was not used in the paper
• HQ-Edit: will be downloaded automatically when starting training
• Visual Genome (VG): obtain the following files
  • images
  • region descriptions
  • question answers

For evaluation, please download the following datasets additionally:

• Open Images v7: test images
• MMEB: download images via

wget https://huggingface.co/datasets/TIGER-Lab/MMEB-eval/resolve/main/images.zip
unzip images.zip -d eval_images/

• ImageNet validation set

Then set the paths to the datasets in ./src/config.py.

Pretrained Models

We provide pretrained FuseLIP models that can be used for evaluation or fine-tuning. The models attain the following performance:

Models can be loaded as follows:

from fuse_clip.fuse_clip_utils import load_model
model, image_processor, text_tokenizer = load_model("chs20/FuseLIP-S-CC3M-MM", device="cuda")

Training

Training scripts are provided for different model variants and datasets and can be run as follows:

SigLIP-S_SF,   SigLIP-B_SF:
SigLIP with score fusion, i.e. arithmetic addition of image embedding + text embedding

# CC3M:
./scripts/train-baseline-cc3m.sh sf [small | base]
# CC12M:
./scripts/train-baseline-cc12m.sh sf [small | base]

SigLIP-S_MLF,   SigLIP-S_MLF:
SigLIP with magiclens fusion - i.e. merging image and text embeddings via a small late fusion module

# CC3M:
./scripts/train-baseline-cc3m.sh mlf [small | base]
# CC12M:
./scripts/train-baseline-cc12m.sh mlf [small | base]

FuseLIP-S:
Our proposed architecture with early fusion of discrete tokens

# CC3M:
./scripts/train-fuselip-cc3m.sh small
# CC12M:
./scripts/train-fuselip-cc12m.sh small

FuseLIP-B:
Our proposed architecture with early fusion of discrete tokens

# CC3M:
./scripts/train-fuselip-cc3m.sh base
# CC12M:
./scripts/train-fuselip-cc12m.sh base

Evaluation

Main evaluation

python src/fuse_eval/eval_all.py

SugarCrepe

To evaluate compositionality performance on SugarCrepe, run:

python src/fuse_eval/eval_sugarcrepe.py

Generating VQA Data from Captions

To generate CC3M-VQA or CC12M-VQA yourself, run

python scripts/generate_vqa_data.py [--model meta-llama/Llama-3.1-8B-Instruct] [--bs 128] [--cc12m]

This script will use all available GPUs, using the specified batch size per device.

Acknowledgement

This codebase gratefully forks from

• OpenCLIP
• TiTok
• VLM2Vec

Citation

If you find this project useful, please cite our paper:

@article{schlarmann2025fuselip,
	title = {FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens},
	author = {Christian Schlarmann and Francesco Croce and Nicolas Flammarion and Matthias Hein},
	year = 2025,
	journal = {arXiv preprint arXiv:2506.03096}
}