ColNetraEmbed

ColNetraEmbed is a state-of-the-art multilingual multimodal embedding model for visual document retrieval, powered by the Gemma3 backbone and using Colbert-style multi-vector representations.

Model Description

ColNetraEmbed is a multilingual multimodal embedding model that encodes documents as multi-vector representations using the ColPali architecture. Each image patch is mapped to a contextualized embedding, enabling fine-grained matching between visual content and text queries through late interaction (MaxSim).

• Model Type: Multilingual Multimodal Embedding Model with ColPali-style Multi-vector representations
• Architecture: ColPali with Gemma3-4B backbone
• Embedding Dimension: 128 per token
• Capabilities: Multilingual, Multimodal (Vision + Text), Multi-vector late interaction
• Use Case: Visual document retrieval, multilingual document understanding, fine-grained visual search

Paper

📄 M3DR: Towards Universal Multilingual Multimodal Document Retrieval

Installation

pip install git+https://github.com/adithya-s-k/colpali.git

Quick Start

import torch
from PIL import Image
from colpali_engine.models import ColGemma3, ColGemmaProcessor3

# Load model and processor
model_name = "Cognitive-Lab/ColNetraEmbed"
model = ColGemma3.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
processor = ColGemmaProcessor3.from_pretrained(model_name)

# Load your images
images = [
    Image.open("document1.jpg"),
    Image.open("document2.jpg"),
]

# Define queries
queries = [
    "What is the total revenue?",
    "Show me the organizational chart",
]

# Process and encode
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)

with torch.no_grad():
    image_embeddings = model(**batch_images)  # Shape: (num_images, num_patches, 128)
    query_embeddings = model(**batch_queries)  # Shape: (num_queries, num_tokens, 128)

# Compute similarity scores using MaxSim
scores = processor.score_multi_vector(
    qs=query_embeddings,
    ps=image_embeddings,
)  # Shape: (num_queries, num_images)

# Get best matches
for i, query in enumerate(queries):
    best_idx = scores[i].argmax().item()
    print(f"Query: '{query}' -> Best match: Image {best_idx + 1} (score: {scores[i, best_idx]:.2f})")

Use Cases

• Document Retrieval: Search through large collections of visual documents
• Visual Question Answering: Answer questions about document content
• Document Understanding: Extract and match information from scanned documents
• Cross-lingual Document Search: Multilingual visual document retrieval

Model Details

• Base Model: Gemma3-4B-IT
• Vision Encoder: SigLIP
• Training Data: Multilingual document datasets
• Embedding Strategy: Multi-vector (Late Interaction)
• Similarity Function: MaxSim (Maximum Similarity)

Performance

ColNetraEmbed achieves strong performance on multilingual document retrieval benchmarks. Evaluated on Nayana-IR Bench (22 languages) and ViDoRe v2.

Benchmark Results

Nayana-IR Cross-Lingual

Nayana-IR Monolingual

ViDoRe v2

Key Results:

• 🏆 Strong multilingual performance with ColBERT-style late interaction
• 📈 124% improvement over ColPali-v1.3 on cross-lingual tasks
• 🌍 Supports 22 languages across diverse script families
• 🔍 Fine-grained matching through token-level MaxSim scoring

Comparison: Multi-vector vs Single-vector

• ColNetraEmbed (multi-vector): More interpretable with token-level attribution
• NetraEmbed (single-vector): Higher accuracy (0.716 vs 0.637) and 250x more efficient storage

See our paper for comprehensive evaluation and architectural comparisons.

Citation

@misc{kolavi2025m3druniversalmultilingualmultimodal,
  title={M3DR: Towards Universal Multilingual Multimodal Document Retrieval}, 
  author={Adithya S Kolavi and Vyoman Jain},
  year={2025},
  eprint={2512.03514},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2512.03514}
}

License

This model is released under the same license as the base Gemma3 model.

Acknowledgments

This work benefited from compute credits for training, inference, and evaluation provided by Modal, acknowledged as a compute sponsor. Dataset curation and synthesis were supported by the Meta LLaMA Impact Grant through our Nayana initiative. We appreciate Meta for continued support of our research efforts at CognitiveLab.

Built on top of the ColPali framework and Gemma3 architecture.