DFN5B-CLIP-ViT-H-14-378 — INT8 Quantized ONNX

INT8 dynamically quantized ONNX export of apple/DFN5B-CLIP-ViT-H-14-378, a ~5B parameter CLIP model trained on Data Filtering Networks (DFN-5B). This quantized variant runs 2.3x faster on CPU with negligible quality loss (cosine similarity 0.985 vs FP32 original).

About the Original Model

DFN5B-CLIP was trained by Apple on 5 billion images filtered from a pool of 43 billion uncurated image-text pairs, using small Data Filtering Networks to automatically curate training data. It achieves 84.2% zero-shot accuracy on ImageNet-1K and 70.9% average across 38 benchmarks.

• Paper: Data Filtering Networks (arXiv:2309.17425)
• Original weights: apple/DFN5B-CLIP-ViT-H-14-378
• Framework: OpenCLIP (originally trained in JAX via AxLearn)

Quantization Details

• Method: Dynamic weight-only INT8 quantization via onnxruntime.quantization.quantize_dynamic with per-channel symmetric scaling. Only MatMul and Gather weights are quantized; activations remain FP32.
• ONNX Runtime version: 1.24.4
• No calibration data required — dynamic quantization determines scales at load time.

Files

Model Specifications

visual_int8.onnx

text_int8.onnx

Inference Timings

CPU — Intel Core i7 (current-gen desktop)

CPU — Intel Core i5 8th Gen

Benchmarks run with ONNX Runtime CPUExecutionProvider, single thread, batch size 1.

Usage

Installation

pip install onnxruntime numpy pillow

Image Embeddings

import numpy as np
import onnxruntime as ort
from PIL import Image

# CLIP preprocessing constants
MEAN = np.array([0.48145466, 0.4578275, 0.40821073], dtype=np.float32)
STD = np.array([0.26862954, 0.26130258, 0.27577711], dtype=np.float32)

def preprocess(image_path, size=378):
    img = Image.open(image_path).convert("RGB")
    w, h = img.size
    side = min(w, h)
    img = img.crop(((w - side) // 2, (h - side) // 2, (w + side) // 2, (h + side) // 2))
    img = img.resize((size, size), Image.BICUBIC)
    arr = (np.array(img, dtype=np.float32) / 255.0 - MEAN) / STD
    return arr.transpose(2, 0, 1)[None]  # (1, 3, H, W)

sess = ort.InferenceSession("visual_int8.onnx", providers=["CPUExecutionProvider"])
pixels = preprocess("your_image.jpg")
image_embeds = sess.run(None, {"pixel_values": pixels})[0]  # (1, 1024)

Text Embeddings

from transformers import CLIPTokenizer

tokenizer = CLIPTokenizer.from_pretrained("apple/DFN5B-CLIP-ViT-H-14-378")
tokens = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="np")

sess = ort.InferenceSession("text_int8.onnx", providers=["CPUExecutionProvider"])
text_embeds = sess.run(None, {"input_ids": tokens["input_ids"]})[0]  # (2, 1024)

Zero-Shot Classification

# Cosine similarity between image and text embeddings
image_embeds = image_embeds / np.linalg.norm(image_embeds, axis=-1, keepdims=True)
text_embeds = text_embeds / np.linalg.norm(text_embeds, axis=-1, keepdims=True)

logit_scale = 14.2849  # from model config
probabilities = (image_embeds @ text_embeds.T * logit_scale).softmax(axis=-1)

Citation

@article{fang2023data,
  title={Data Filtering Networks},
  author={Fang, Alex and Jose, Albin Madappally and Jain, Amit and Schmidt, Ludwig and Toshev, Alexander and Shankar, Vaishaal},
  journal={arXiv preprint arXiv:2309.17425},
  year={2023}
}