Unpaired Rosetta embeddings

This dataset holds the precomputed image and text embeddings behind the vision-language experiments of the paper Shared Geometry as a Rosetta Stone: Cross-Modal Alignment Without Paired Data. The code in github.com/dominik-schnaus/unpaired-rosetta downloads the files it needs from here, and the notebook unpaired_rosetta.ipynb in that repository does it automatically.

Contents

The embeddings come from 7 vision models and 3 language models:

• Vision: DINOv3 7B/16, Franca G/14 (CLS token and mean pooling), DINOv2 G/14 and B/14, iBOT Swin-T and iBOT B/16.
• Language: MPNet, Qwen3-Embedding-8B, and Qwen3-8B with generative token pooling.

They cover the following datasets:

• Image-caption datasets: MS COCO 2014 (train and val), Stanford Paragraph Captioning, DCI and DOCCI.
• Zero-shot classification: CIFAR-10, CIFAR-100 and ImageNet-100, with their labels and the class names written into prompts.

File format

Every file is a PyTorch tensor that you can load with torch.load, and the paths follow embeddings/<dataset>/<modality>/<model>.pt.

• Image embeddings are bfloat16 tensors of shape [images, dim].
• Caption embeddings are bfloat16 tensors of shape [images, captions, dim]. Images with fewer captions than the maximum are padded with NaN.
• Class prompt embeddings are in language_<prompt hash>/ folders with shape [classes, prompts, dim].
• labels.pt holds the integer class of each image.

Row i of an image file and row i of a caption file of the same dataset belong to the same image.

License

The embeddings are computed from the datasets listed above, so their use is subject to the license of each source dataset and model.

Citation

@article{schnaus2026shared,
  title={Shared Geometry as a Rosetta Stone: Cross-Modal Alignment Without Paired Data},
  author={Schnaus, Dominik and Dag{\`e}s, Thomas and Cremers, Daniel and Wang, Xi and Isola, Phillip},
  journal={arXiv preprint arXiv:2610.09411},
  year={2026}
}