QuanSun/EVA-CLIP

Model

117

stars

34

commits

10

repos using this model

6

linked in READMEs

Jun 14, 2023

updated

README

https://github.com/baaivision/EVA/tree/master/EVA-CLIP

Summary of EVA-CLIP performance

summary_tab

Model Card

EVA-01-CLIP Series (MIM teacher: OpenAI CLIP-Large)

model name	total #params	training precision	training data	training batch size	gpus for training	IN-1K zero-shot top-1	MSCOCO T2I R@5	weight
`EVA01_CLIP_g_14_psz14_s11B`	1.1B	`fp16`	LAION-400M	41K	256 A100(40GB)	78.5	68.5	🤗 HF link (`2.2GB`)
`EVA01_CLIP_g_14_plus_psz14_s11B`	1.3B	`fp16`	Merged-2B	114K	112 A100(40GB)	79.3	74.0	🤗 HF link (`2.7GB`)

EVA-02-CLIP Series (MIM teacher: `EVA01_CLIP_g_14_psz14_s11B`)

model name	image enc. init. ckpt	text enc. init. ckpt	total #params	training precision	training data	training batch size	gpus for training	IN-1K zero-shot top-1	MSCOCO T2I R@5	weight
`EVA02_CLIP_B_psz16_s8B`	`EVA02_B_psz14to16`	`openai/clip-vit-base-patch16`	149M	`fp16`	Merged-2B	131K	64 A100(40GB)	74.7	66.9	🤗 HF link (`300MB`)
`EVA02_CLIP_L_psz14_s4B`	`EVA02_L_psz14`	`openai/clip-vit-large-patch14`	428M	`fp16`	Merged-2B	131K	128 A100(40GB)	79.8	71.2	🤗 HF link (`856MB`)
`EVA02_CLIP_L_336_psz14_s6B`	`EVA02_CLIP_L_psz14_224to336`	`EVA02_CLIP_L_psz14_224to336`	428M	`fp16`	Merged-2B	61K	128 A100(40GB)	80.4	71.7	🤗 HF link (`856MB`)
`EVA02_CLIP_E_psz14_s4B.pt`	`EVA02_E_psz14`	`laion/CLIP-ViT-H-14-laion2B-s32B-b79K`	4.7B	`fp16`	LAION-2B	115K	144 A100(80GB)	81.9	74.7	🤗 HF link (`9.4GB`)
`EVA02_CLIP_E_psz14_plus_s9B.pt`	`EVA02_E_psz14`	`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`	5.0B	`bf16`	LAION-2B	144K	144 A100(80GB)	82.0	75.0	🤗 HF link (`10.1GB`)

To construct Merged-2B, we merged 1.6 billion samples from LAION-2B dataset with 0.4 billion samples from COYO-700M.
To our knowledge, EVA-CLIP series are the most performant open-sourced CLIP models at all scales, evaluated via zero-shot classification performance, especially on mainstream classification benchmarks such as ImageNet along with its variants. For more details about EVA-CLIP, please refer to our paper.

Pretrained

model name	total #params	training precision	download link
`EVA01_g_psz14`	1.0B	`fp16`	🤗 HF link (`2.0GB`)
`EVA02_B_psz14to16`	86M	`fp16`	🤗 HF link (`176MB`)
`EVA02_L_psz14`	304M	`fp16`	🤗 HF link (`609MB`)
`EVA02_CLIP_L_psz14_224to336`	428M	`fp16`	🤗 HF link (`857MB`)
`EVA02_E_psz14`	4.4B	`fp16`	🤗 HF link (`8.7GB`)
`openai/clip-vit-base-patch16`	149M	`fp16`	🤗 HF link (`599MB`)
`openai/clip-vit-large-patch14`	428M	`fp16`	🤗 HF link (`1.7GB`)
`laion/CLIP-ViT-H-14-laion2B-s32B-b79K`	1.0B	`bf16`	🤗 HF link (`3.9GB`)
`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`	1.8B	`bf16`	🤗 HF link part1 part2(`9.9GB`+`169M`)

EVA02_B_psz14to16 interpolates the kernel size of patch_embed from 14x14 to 16x16, and interpolate the pos_embed from 16x16 to 14x14.
EVA02_CLIP_L_psz14_224to336 interpolates the pos_embed from 16x16 to 24x24 for training EVA02_CLIP_L_336_psz14_s6B.
laion/CLIP-ViT-bigG-14-laion2B-39B-b160k consists of 2 parts of weights, part1 and part2.

Contributors

33 commits

1 commits

Similar repos

BAAI/EVA-CLIP-18B

17

BAAI/EVA-CLIP-8B

51

BAAI/EVA

30

laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg-soup

25

timm/eva02_large_patch14_448.mim_m38m_ft_in1k

14

laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft-soup

22

imirandam/CLIP_COCO

0

baaivision/EVA

EVA Series: Visual Representation Fantasies from BAAI

2,695

Used in code

tonera/vitoom

AI productivity tools

9

xxw77777kk/TextVLoc

5

Justin21523/3d-animation-lora-pipeline

A full pipeline for 3D character processing — including video-to-frames extraction, layered…

1

tjluyao/fastSD

1

niketp03/taskpriors

0

ARIJIT00171/RE-TRIANGLE

Reproducing the results from TRAINGLE paper.

0

wh1te125/edit_mind

0

hirunima/DGE

DGE - A Dynamic Metric and Grounded Evaluation Benchmarks for Text-to-Image and Image Editing Models

0

anon-code-ai/anon-triangle

0

ThiBuiPhamMinh/ClipSumary

0

Linked in READMEs

openai/CLIP

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an…

34,266

baaivision/EVA

EVA Series: Visual Representation Fantasies from BAAI

2,695

kakaobrain/coyo-dataset

COYO-700M: Large-scale Image-Text Pair Dataset

1,254

BAAI-DCAI/Bunny

A family of lightweight multimodal models.

1,052

smthemex/ComfyUI_StoryDiffusion

You can using StoryDiffusion in ComfyUI

516

wswszhys/ComfyUI_StoryDiffusion

1

QuanSun/EVA-CLIP

Model

117

stars

34

commits

10

repos using this model

6

linked in READMEs

Jun 14, 2023

updated

README

https://github.com/baaivision/EVA/tree/master/EVA-CLIP

Summary of EVA-CLIP performance

summary_tab

Model Card

EVA-01-CLIP Series (MIM teacher: OpenAI CLIP-Large)

model name	total #params	training precision	training data	training batch size	gpus for training	IN-1K zero-shot top-1	MSCOCO T2I R@5	weight
`EVA01_CLIP_g_14_psz14_s11B`	1.1B	`fp16`	LAION-400M	41K	256 A100(40GB)	78.5	68.5	🤗 HF link (`2.2GB`)
`EVA01_CLIP_g_14_plus_psz14_s11B`	1.3B	`fp16`	Merged-2B	114K	112 A100(40GB)	79.3	74.0	🤗 HF link (`2.7GB`)

EVA-02-CLIP Series (MIM teacher: `EVA01_CLIP_g_14_psz14_s11B`)

model name	image enc. init. ckpt	text enc. init. ckpt	total #params	training precision	training data	training batch size	gpus for training	IN-1K zero-shot top-1	MSCOCO T2I R@5	weight
`EVA02_CLIP_B_psz16_s8B`	`EVA02_B_psz14to16`	`openai/clip-vit-base-patch16`	149M	`fp16`	Merged-2B	131K	64 A100(40GB)	74.7	66.9	🤗 HF link (`300MB`)
`EVA02_CLIP_L_psz14_s4B`	`EVA02_L_psz14`	`openai/clip-vit-large-patch14`	428M	`fp16`	Merged-2B	131K	128 A100(40GB)	79.8	71.2	🤗 HF link (`856MB`)
`EVA02_CLIP_L_336_psz14_s6B`	`EVA02_CLIP_L_psz14_224to336`	`EVA02_CLIP_L_psz14_224to336`	428M	`fp16`	Merged-2B	61K	128 A100(40GB)	80.4	71.7	🤗 HF link (`856MB`)
`EVA02_CLIP_E_psz14_s4B.pt`	`EVA02_E_psz14`	`laion/CLIP-ViT-H-14-laion2B-s32B-b79K`	4.7B	`fp16`	LAION-2B	115K	144 A100(80GB)	81.9	74.7	🤗 HF link (`9.4GB`)
`EVA02_CLIP_E_psz14_plus_s9B.pt`	`EVA02_E_psz14`	`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`	5.0B	`bf16`	LAION-2B	144K	144 A100(80GB)	82.0	75.0	🤗 HF link (`10.1GB`)

To construct Merged-2B, we merged 1.6 billion samples from LAION-2B dataset with 0.4 billion samples from COYO-700M.
To our knowledge, EVA-CLIP series are the most performant open-sourced CLIP models at all scales, evaluated via zero-shot classification performance, especially on mainstream classification benchmarks such as ImageNet along with its variants. For more details about EVA-CLIP, please refer to our paper.

Pretrained

model name	total #params	training precision	download link
`EVA01_g_psz14`	1.0B	`fp16`	🤗 HF link (`2.0GB`)
`EVA02_B_psz14to16`	86M	`fp16`	🤗 HF link (`176MB`)
`EVA02_L_psz14`	304M	`fp16`	🤗 HF link (`609MB`)
`EVA02_CLIP_L_psz14_224to336`	428M	`fp16`	🤗 HF link (`857MB`)
`EVA02_E_psz14`	4.4B	`fp16`	🤗 HF link (`8.7GB`)
`openai/clip-vit-base-patch16`	149M	`fp16`	🤗 HF link (`599MB`)
`openai/clip-vit-large-patch14`	428M	`fp16`	🤗 HF link (`1.7GB`)
`laion/CLIP-ViT-H-14-laion2B-s32B-b79K`	1.0B	`bf16`	🤗 HF link (`3.9GB`)
`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`	1.8B	`bf16`	🤗 HF link part1 part2(`9.9GB`+`169M`)

EVA02_B_psz14to16 interpolates the kernel size of patch_embed from 14x14 to 16x16, and interpolate the pos_embed from 16x16 to 14x14.
EVA02_CLIP_L_psz14_224to336 interpolates the pos_embed from 16x16 to 24x24 for training EVA02_CLIP_L_336_psz14_s6B.
laion/CLIP-ViT-bigG-14-laion2B-39B-b160k consists of 2 parts of weights, part1 and part2.

Similar repos

BAAI/EVA-CLIP-18B

17

BAAI/EVA-CLIP-8B

51

BAAI/EVA

30

laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg-soup

25

timm/eva02_large_patch14_448.mim_m38m_ft_in1k

14

laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft-soup

22

imirandam/CLIP_COCO

0

baaivision/EVA

EVA Series: Visual Representation Fantasies from BAAI

2,695

Used in code

tonera/vitoom

AI productivity tools

9

xxw77777kk/TextVLoc

5

Justin21523/3d-animation-lora-pipeline

A full pipeline for 3D character processing — including video-to-frames extraction, layered…

1

tjluyao/fastSD

1

niketp03/taskpriors

0

ARIJIT00171/RE-TRIANGLE

Reproducing the results from TRAINGLE paper.

0

wh1te125/edit_mind

0

hirunima/DGE

DGE - A Dynamic Metric and Grounded Evaluation Benchmarks for Text-to-Image and Image Editing Models

0

anon-code-ai/anon-triangle

0

ThiBuiPhamMinh/ClipSumary

0

Linked in READMEs

openai/CLIP

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an…

34,266

baaivision/EVA

EVA Series: Visual Representation Fantasies from BAAI

2,695

kakaobrain/coyo-dataset

COYO-700M: Large-scale Image-Text Pair Dataset

1,254

BAAI-DCAI/Bunny

A family of lightweight multimodal models.

1,052

smthemex/ComfyUI_StoryDiffusion

You can using StoryDiffusion in ComfyUI

516

wswszhys/ComfyUI_StoryDiffusion

1

Contributors

33 commits

1 commits