NetraEmbed

NetraEmbed is a state-of-the-art multilingual multimodal embedding model for visual document retrieval with Matryoshka representation learning, powered by the Gemma3 backbone.

Model Description

NetraEmbed is a multilingual multimodal embedding model that encodes both visual documents and text queries into single dense vectors. It supports multiple languages and enables efficient similarity search at multiple embedding dimensions (768, 1536, 2560) through Matryoshka representation learning.

• Model Type: Multilingual Multimodal Embedding Model with Matryoshka embeddings
• Architecture: BiEncoder with Gemma3-4B backbone
• Embedding Dimensions: 768, 1536, 2560 (Matryoshka)
• Capabilities: Multilingual, Multimodal (Vision + Text)
• Use Case: Visual document retrieval, multilingual semantic search, cross-lingual document understanding

Paper

📄 M3DR: Towards Universal Multilingual Multimodal Document Retrieval

Installation

pip install git+https://github.com/adithya-s-k/colpali.git

Quick Start

import torch
from PIL import Image
from colpali_engine.models import BiGemma3, BiGemmaProcessor3

# Load model and processor
model_name = "Cognitive-Lab/NetraEmbed"

# Load model once (supports all Matryoshka dimensions)
model = BiGemma3.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
processor = BiGemmaProcessor3.from_pretrained(model_name)

# Load your images
images = [
    Image.open("document1.jpg"),
    Image.open("document2.jpg"),
]

# Define queries
queries = [
    "What is the total revenue?",
    "Show me the organizational chart",
]

# Process and encode
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_texts(queries).to(model.device)

# Choose embedding dimension at inference time: 768, 1536, or 2560
# Use lower dims for faster search, higher for better accuracy
embedding_dim = 1536  # Balanced performance

with torch.no_grad():
    image_embeddings = model(**batch_images, embedding_dim=embedding_dim)  # Shape: (num_images, embedding_dim)
    query_embeddings = model(**batch_queries, embedding_dim=embedding_dim)  # Shape: (num_queries, embedding_dim)

# Compute similarity scores using cosine similarity
scores = processor.score(
    qs=query_embeddings,
    ps=image_embeddings,
)  # Shape: (num_queries, num_images)

# Get best matches
for i, query in enumerate(queries):
    best_idx = scores[i].argmax().item()
    print(f"Query: '{query}' -> Best match: Image {best_idx + 1} (score: {scores[i, best_idx]:.4f})")

Testing Multiple Dimensions

You can test different embedding dimensions without reloading the model:

# Load model once
model = BiGemma3.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)

# Test all Matryoshka dimensions
for embedding_dim in [768, 1536, 2560]:
    print(f"\nTesting dimension: {embedding_dim}")

    with torch.no_grad():
        image_embeddings = model(**batch_images, embedding_dim=embedding_dim)
        query_embeddings = model(**batch_queries, embedding_dim=embedding_dim)

    scores = processor.score(qs=query_embeddings, ps=image_embeddings)
    print(f"Scores shape: {scores.shape}")
    print(f"Best match score: {scores.max().item():.4f}")

Matryoshka Embeddings

NetraEmbed supports three embedding dimensions that can be selected at inference time:

Key Advantage: Load the model once and dynamically choose dimensions at inference time. No need to reload the model to test different dimensions or switch between accuracy/speed trade-offs!

Use Cases

• Efficient Document Retrieval: Fast search through millions of documents
• Semantic Search: Find visually similar documents
• Scalable Vector Search: Works with FAISS, Milvus, Pinecone, etc.
• Cross-lingual Retrieval: Multilingual visual document search

Model Details

• Base Model: Gemma3-4B-IT
• Vision Encoder: SigLIP
• Training Data: Multilingual document datasets
• Embedding Strategy: Single-vector (BiEncoder)
• Similarity Function: Cosine similarity
• Matryoshka Dimensions: 768, 1536, 2560

Performance

NetraEmbed achieves state-of-the-art performance on multilingual document retrieval benchmarks. Evaluated on Nayana-IR Bench (22 languages) and ViDoRe v2.

Benchmark Results

Nayana-IR Cross-Lingual

Nayana-IR Monolingual

ViDoRe v2

Key Results:

• 🏆 State-of-the-art on multilingual retrieval (0.716 NDCG@5 cross-lingual)
• 📈 152% improvement over ColPali-v1.3 on cross-lingual tasks
• 🌍 Consistent performance across 22 languages and diverse scripts
• ⚡ 250x more efficient than multi-vector approaches (~10KB vs ~2.5MB per document)

See our paper for comprehensive evaluation and per-language analysis.

Citation

@misc{kolavi2025m3druniversalmultilingualmultimodal,
  title={M3DR: Towards Universal Multilingual Multimodal Document Retrieval}, 
  author={Adithya S Kolavi and Vyoman Jain},
  year={2025},
  eprint={2512.03514},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2512.03514}
}

License

This model is released under the same license as the base Gemma3 model.

Acknowledgments

This work benefited from compute credits for training, inference, and evaluation provided by Modal, acknowledged as a compute sponsor. Dataset curation and synthesis were supported by the Meta LLaMA Impact Grant through our Nayana initiative. We appreciate Meta for continued support of our research efforts at CognitiveLab.

Built on top of the Gemma3 architecture with Matryoshka representation learning.