CLIP Embedder Library

A Rust library for generating CLIP embeddings from images and text using the Candle framework.

Features

• Easy-to-use API: Simple struct-based interface with new() constructor and embedding methods
• Image embeddings: Generate 512-dimensional embeddings from image files, DynamicImages, or raw bytes
• Text embeddings: Generate 512-dimensional embeddings from text strings
• Batch processing: Process multiple images efficiently in a single call
• GPU acceleration: Automatic Metal (macOS) or CUDA support with CPU fallback
• Model management: Automatic download and caching of CLIP models from HuggingFace

Quick Start

Basic Usage

use anyhow::Result;
use clipper::ClipEmbedder;

fn main() -> Result<()> {
    // Initialize the CLIP embedder (downloads model on first run)
    let embedder = ClipEmbedder::new(None, None, false)?;
    
    // Get image embedding
    let image_embedding = embedder.get_image_embedding("path/to/image.jpg")?;
    println!("Image embedding length: {}", image_embedding.len()); // 512
    
    // Get text embedding  
    let text_embedding = embedder.get_text_embedding("a photo of a cat")?;
    println!("Text embedding length: {}", text_embedding.len()); // 512
    
    Ok(())
}

API Reference

ClipEmbedder

The main struct that provides access to CLIP embeddings.

Constructor

ClipEmbedder::new(
    model_path: Option<String>,      // Optional custom model path
    tokenizer_path: Option<String>,  // Optional custom tokenizer path  
    use_cpu: bool                    // Force CPU usage if true
) -> Result<ClipEmbedder>

Parameters:

• model_path: Path to a local model file. If None, downloads from HuggingFace.
• tokenizer_path: Path to a local tokenizer file. If None, downloads from HuggingFace.
• use_cpu: Set to true to force CPU usage, false to use GPU if available.

Methods

get_image_embedding()

fn get_image_embedding(&self, image_path: &str) -> Result<Vec<f32>>

Generates a 512-dimensional embedding vector for an image file.

Parameters:

• image_path: Path to the image file (supports common formats: JPG, PNG, etc.)

Returns: Vec<f32> with 512 elements representing the image embedding.

get_image_embedding_from_dynamic()

fn get_image_embedding_from_dynamic(&self, image: image::DynamicImage) -> Result<Vec<f32>>

Generates a 512-dimensional embedding vector from a DynamicImage (from the image crate).

Parameters:

• image: A DynamicImage instance that will be resized to the model's required size

Returns: Vec<f32> with 512 elements representing the image embedding.

get_image_embedding_from_bytes()

fn get_image_embedding_from_bytes(&self, image_bytes: &[u8]) -> Result<Vec<f32>>

Generates a 512-dimensional embedding vector from raw image bytes.

Parameters:

• image_bytes: Raw bytes of an image file (PNG, JPEG, etc.) that will be decoded and resized

Returns: Vec<f32> with 512 elements representing the image embedding.

get_text_embedding()

fn get_text_embedding(&self, text: &str) -> Result<Vec<f32>>

Generates a 512-dimensional embedding vector for a text string.

Parameters:

• text: The input text string to encode

Returns: Vec<f32> with 512 elements representing the text embedding.

Batch Processing Methods

get_image_embeddings()

fn get_image_embeddings(&self, image_paths: &[String]) -> Result<Vec<Vec<f32>>>

Generates 512-dimensional embedding vectors for multiple image files efficiently.

Parameters:

• image_paths: Slice of image file paths

Returns: Vec<Vec<f32>> where each inner vector contains 512 elements. Order matches input.

get_image_embeddings_from_dynamic()

fn get_image_embeddings_from_dynamic(&self, images: Vec<image::DynamicImage>) -> Result<Vec<Vec<f32>>>

Generates 512-dimensional embedding vectors for multiple DynamicImage instances.

Parameters:

• images: Vector of DynamicImage instances

Returns: Vec<Vec<f32>> where each inner vector contains 512 elements. Order matches input.

get_image_embeddings_from_bytes()

fn get_image_embeddings_from_bytes(&self, image_bytes_list: &[&[u8]]) -> Result<Vec<Vec<f32>>>

Generates 512-dimensional embedding vectors for multiple images from raw bytes.

Parameters:

• image_bytes_list: Slice of byte slices, each containing raw image data

Returns: Vec<Vec<f32>> where each inner vector contains 512 elements. Order matches input.

Example: Computing Similarity

use anyhow::Result;
use clipper::ClipEmbedder;

fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
    let dot_product: f32 = a.iter().zip(b.iter()).map(|(x, y)| x * y).sum();
    let norm_a: f32 = a.iter().map(|x| x * x).sum::<f32>().sqrt();
    let norm_b: f32 = b.iter().map(|x| x * x).sum::<f32>().sqrt();
    
    if norm_a == 0.0 || norm_b == 0.0 {
        0.0
    } else {
        dot_product / (norm_a * norm_b)
    }
}

fn main() -> Result<()> {
    let embedder = ClipEmbedder::new(None, None, false)?;
    
    // Compare image and text
    let image_embedding = embedder.get_image_embedding("assets/cat.jpg")?;
    let text_embedding = embedder.get_text_embedding("a photo of a cat")?;
    
    let similarity = cosine_similarity(&image_embedding, &text_embedding);
    println!("Image-text similarity: {:.4}", similarity);
    
    Ok(())
}

Example: Batch Processing

use anyhow::Result;
use clipper::ClipEmbedder;

fn main() -> Result<()> {
    let embedder = ClipEmbedder::new(None, None, false)?;
    
    // Process multiple images at once (more efficient than individual calls)
    let image_paths = vec![
        "assets/cat1.jpg".to_string(),
        "assets/cat2.jpg".to_string(),
        "assets/dog1.jpg".to_string(),
    ];
    
    let batch_embeddings = embedder.get_image_embeddings(&image_paths)?;
    println!("Processed {} images", batch_embeddings.len());
    
    // Each embedding is 512 dimensions
    for (i, embedding) in batch_embeddings.iter().enumerate() {
        println!("Image {}: {} dimensions", i + 1, embedding.len());
    }
    
    // Also works with DynamicImages and raw bytes
    let dynamic_images = vec![
        image::open("assets/image1.jpg")?,
        image::open("assets/image2.jpg")?,
    ];
    let dynamic_batch = embedder.get_image_embeddings_from_dynamic(dynamic_images)?;
    
    Ok(())
}

use anyhow::Result;
use clipper::ClipEmbedder;
use std::fs;

fn main() -> Result<()> {
    let embedder = ClipEmbedder::new(None, None, false)?;
    
    // Method 1: From file path
    let embedding1 = embedder.get_image_embedding("assets/image.jpg")?;
    
    // Method 2: From DynamicImage (useful when you already have an image loaded)
    let dynamic_image = image::open("assets/image.jpg")?;
    let embedding2 = embedder.get_image_embedding_from_dynamic(dynamic_image)?;
    
    // Method 3: From raw bytes (useful for web uploads, database blobs, etc.)
    let image_bytes = fs::read("assets/image.jpg")?;
    let embedding3 = embedder.get_image_embedding_from_bytes(&image_bytes)?;
    
    // All methods produce identical results
    assert_eq!(embedding1.len(), 512);
    assert_eq!(embedding2.len(), 512);
    assert_eq!(embedding3.len(), 512);
    
    Ok(())
}

Command Line Interface

The library also includes a CLI tool for testing:

# Use default images and text
cargo run

# Use custom images
cargo run -- --images image1.jpg,image2.jpg

# Use custom text sequences
cargo run -- --sequences "a cat","a dog","a bird"

# Force CPU usage
cargo run -- --cpu

# Use custom model files
cargo run -- --model /path/to/model.safetensors --tokenizer /path/to/tokenizer.json

Model Information

This library uses the CLIP ViT-Base-Patch32 model by default:

• Model: openai/clip-vit-base-patch32
• Embedding size: 512 dimensions
• Image input size: 224x224 pixels (automatically resized)
• Text context length: Up to 77 tokens

Performance Notes

• First run: Downloads ~400MB model files from HuggingFace (cached locally)
• GPU acceleration: Automatically uses Metal (macOS) or CUDA if available
• Memory usage: ~2GB GPU memory for inference
• Speed: ~10-50ms per embedding depending on hardware

Dependencies

• candle-core: Tensor operations and model loading
• candle-transformers: CLIP model implementation
• tokenizers: Text tokenization
• image: Image loading and preprocessing
• hf-hub: HuggingFace model downloading

License

This project uses the same license as the underlying Candle framework.