ruclip-vit-base-patch16-384

RuCLIP (Russian Contrastive Language–Image Pretraining) is a multimodal model for obtaining images and text similarities and rearranging captions and pictures. RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and multimodal learning.

Model was trained by Sber AI and SberDevices teams.

• Task: text ranking; image ranking; zero-shot image classification;
• Type: encoder
• Num Parameters: 150M
• Training Data Volume: 240 million text-image pairs
• Language: Russian
• Context Length: 77
• Transformer Layers: 12
• Transformer Width: 512
• Transformer Heads: 8
• Image Size: 384
• Vision Layers: 12
• Vision Width: 768
• Vision Patch Size: 16

Usage Github

pip install ruclip

clip, processor = ruclip.load("ruclip-vit-base-patch16-384", device="cuda")

Performance

We have evaluated the performance on the following datasets:

Authors

• Alex Shonenkov: Github, Kaggle GM
• Daniil Chesakov: Github
• Denis Dimitrov: Github
• Igor Pavlov: Github