Multimodal Image Search with Cross-Encoder Reranking on the Amazon Berkeley Objects Dataset

A text-to-image search engine over the Amazon Berkeley Objects (ABO) dataset, with an offline evaluation pipeline that measures whether a cross-encoder reranking stage actually improves retrieval quality — and at what latency cost. CLIP performs efficient vector retrieval over ~400k images, while a vision-language cross-encoder examines each query-image pair jointly, allowing it to reorder candidates using richer multimodal reasoning than cosine similarity alone.

TL;DR: CLIP ViT-B/32 retrieves candidates from Qdrant; Qwen3-VL-Reranker-2B reorders them. Evaluated via VLM-judged graded relevance rather than exact-match retrieval, using a pooling-based methodology inspired by TREC's approach to IR evaluation — judge a union of candidate pools from multiple systems rather than exhaustively labeling the full corpus, which is standard practice for evaluating large-scale retrieval systems where labeling every item is infeasible. Reranking improves every rank-sensitive metric (MAP +0.040, MRR +0.042, nDCG@5 +0.070) — see Results.

---

Table of contents

• Architecture
• Dataset
• Evaluation methodology
• Results
• Limitations
• Repo structure
• Setup
• Reproducing Results

---

Architecture

Search pipeline (main.py)

Query
  │
  ▼
CLIP ViT-B/32 (text encoder)
  │
  ▼
Top-100 retrieval from Qdrant
  │
  ▼
Qwen3-VL-Reranker-2B (cross-encoder, reranks top-20)
  │
  ▼
Final ranking

Served via FastAPI.

• Retriever: openai/clip-vit-base-patch32
• Reranker: Qwen/Qwen3-VL-Reranker-2B
• Vector DB: Qdrant Cloud (collection: images)

Evaluation pipeline

Laptop
  │  upload dataset once (~3.5 GB)
  ▼
Modal Volume (abo-images)
  │
  ▼
build_pool.py   (Modal GPU)   →  pool.csv
  │
  ▼
judge.py        (Modal GPU)   →  judgements.csv
  │
  ▼
compute_metrics.py  (local CPU)  →  Precision@K / Recall@K / MRR / MAP / nDCG@K
  │
  ▼
build_clip_baseline_pool.py (local CPU)  →  clip_pool.csv → compute_metrics.py again
  │
  ▼
latency_benchmark.py (Modal GPU)  →  latency_results.csv, latency_summary.csv

Everything after the initial dataset upload runs on Modal, except the CLIP-baseline comparison and metrics computation, which are cheap enough to run without GPU.

---

Dataset

Amazon Berkeley Objects (ABO) — ~398210 product images.

Benchmark construction: 500 images randomly sampled; for each, Qwen2.5-VL-3B-Instruct generated three query types (basic, attribute, natural) describing the image → benchmark_queries.csv. In practice this yielded 1,270 unique query strings, not the theoretical 1,500 (500×3) — due to duplication, not missing data (see Limitations for the breakdown).

---

Evaluation methodology

We evaluate graded relevance, not exact-image retrieval — many visually similar products (same item/different angle, same category/different color) can equally satisfy a query, so exact-match would understate real search quality.

The ground-truth construction follows TREC-style pooling: rather than exhaustively labeling relevance for every image in the ~400k-image corpus against every query (infeasible), we build a pool of candidates per query by taking the union of what CLIP and CLIP+reranker each retrieve, and judge relevance only within that pool. This is the standard approach used in IR evaluation at scale (TREC conferences have used pooling since the 1990s for exactly this reason) — it makes large-corpus evaluation tractable while still producing valid, standard IR metrics, at the cost of the pool-relative caveats discussed in Limitations.

Stage 1 — Candidate pool (build_pool.py)

For every benchmark query: CLIP top-100 retrieval → Qwen3-VL-Reranker rerank → union of both → pool.csv (query, gt_image, candidate_image, candidate_path).

Stage 2 — Relevance judging (judge.py)

Every unique (query, candidate_image) pair judged by Qwen2.5-VL-3B-Instruct on a 0/1/2 scale → judgements.csv.

Pool capped to the top 25 candidates per query (by rerank score) before judging, to bound VLM compute cost (~11.5h on a single A10G at full depth would otherwise have been ~46h).

Stage 3 — Metrics (compute_metrics.py)

Precision@K, Recall@K, MRR, MAP, nDCG@K — computed per query and averaged, with an optional breakdown by query type.

CLIP-only baseline (build_clip_baseline_pool.py)

Re-ranks the same already-judged candidates by raw CLIP similarity (instead of rerank score), reusing compute_metrics.py unchanged for an apples-to-apples comparison — without re-running the expensive VLM judge.

---

Results

Accuracy — 1,270 unique queries, top-25 candidates/query judged, 31,750 total judged pairs:

Reranking gives a consistent, real improvement across every rank-sensitive metric, largest at shallow K (nDCG@5: +0.070) and shrinking as K grows (nDCG@20: +0.043) — consistent with reranking's job of pulling the best items to the very top rather than changing which items are broadly relevant.

Full per-query breakdowns: per_query_metrics.csv (reranker), per_query_metrics_clip_baseline.csv (CLIP baseline).

Latency (A10G, TOP_K=100, rerank num_candidates=20 — matching production defaults in main.py; n=100 sampled queries):

Reranking adds ~613 ms mean latency per query (+250% over CLIP-only) — roughly 3.5x slower end to end. It's also far less consistent: CLIP-only varies by ~16 ms across its full min–max range, while CLIP+Reranker varies by over 1000 ms (603–1687 ms), and p99 (1227 ms) sits well above the mean. This is the expected shape for a cross-encoder scoring a variable-size batch of images per request, but it means tail latency, not average latency, is the real cost of reranking for a user-facing search endpoint — worth keeping in mind against the accuracy gains in the table above when deciding whether reranking should run on every request or only selectively.

---

Limitations

• Recall@K is pool-relative, not corpus-relative — the expected tradeoff of the TREC-style pooling approach described above. Only the top-25 candidates per query were judged, so Recall@K reflects "of the relevant items found within the judged pool," not recall against the full ~400k image corpus.
• CLIP-baseline metrics are a slight upper bound. The judged pool was built from the reranker's top-25, so CLIP-baseline is only scored on candidates the reranker already approved of. A candidate CLIP would rank highly but the reranker demoted out of the top-25 has no relevance label and is excluded. CLIP/reranker top-25s likely overlap heavily in practice, so this bias is believed small, but hasn't been directly measured — doing so would require judging the union of CLIP-top-25 and reranker-top-25 per query, not just the reranker's top-25.
• 1,270 unique queries, not 1,500. Entirely duplication, not missing data (0 NaNs across all three columns) — concentrated in the basic query type (340/500 unique, 32% duplicated), with attribute (472/500) and natural (490/500) far less affected. Expected: basic queries are short, generic descriptions ("a black phone case") with a much smaller possible-string space across 500 products than attribute/natural queries — a property of the query-generation process, not a pipeline bug.

---

Repo structure

.
├── main.py                       # FastAPI search prototype (CLIP + reranker)
├── build_pool.py                 # Stage 1 — Modal GPU job, builds pool.csv
├── build_clip_baseline_pool.py   # CLIP-only baseline pool builder
├── judge.py                      # Stage 2 — Modal GPU job, builds judgements.csv
├── latency_benchmark.py          # Modal GPU job, CLIP vs CLIP+reranker latency
├── benchmark_queries.csv         # benchmark query set
├── pool.csv                      # pooled candidates for judging
├── clip_pool.csv                 # CLIP-only baseline candidate pool
├── judgements.csv                # VLM relevance judgments
├── helper-scripts/
│   ├── compute_metrics.py        # IR metric computation
│   ├── upsert_to_qdrant.py       # bulk Qdrant upload helper
│   ├── upload_data_to_modal.py   # dataset upload helper
│   ├── inspect_queries.py        # query inspection utility
│   ├── generate_clip_embeddings.ipynb
│   ├── bencharmaking_dataset.ipynb
│   └── check-data.ipynb
├── results/
│   ├── per_query_metrics.csv
│   ├── per_query_metrics_clip_baseline.csv
│   └── latency_summary.csv
└── LICENSE

Modal Volume layout (abo-images)

/images
├── small/                    # product images
├── metadata/
│   └── images.csv.gz         # image_id -> relative path
├── benchmark_queries.csv     # image_id, basic, attribute, natural
├── pool.csv                  # Stage 1 output
├── judgements_partial.csv    # Stage 2 checkpoint
├── judgements.csv            # Stage 2 output
├── failed_judge_rows.csv     # Stage 2 errors / unparsed VLM output
├── latency_results.csv       # per-query latency timings
└── latency_summary.csv       # aggregate latency stats

---

Setup

1. Create the Modal secret for Qdrant credentials:

modal secret create qdrant-secret QDRANT_URL=... QDRANT_API_KEY=...

2. Upload the dataset (~3.5 GB) to a Modal Volume named abo-images, matching the layout above.
3. Install local dependencies (for the CPU-only steps):

pip install pandas numpy tqdm scikit-learn qdrant-client

⚠️ Rotate the Qdrant API key before making this repo public — it was previously committed in plaintext in an early version of main.py.

---

Reproducing Results

Run the pipeline in order:

# Stage 1 — build the candidate pool (Modal, GPU)
modal run build_pool.py

# Stage 2 — judge relevance (Modal, GPU) — test on a subset first
modal run judge.py --limit 20
modal run judge.py

# Pull results locally
modal volume get abo-images judgements.csv .
modal volume get abo-images pool.csv .
modal volume get abo-images benchmark_queries.csv .

# Stage 3 — compute metrics for CLIP + reranker (local/Colab, CPU)
python helper-scripts/compute_metrics.py --judgements judgements.csv --pool pool.csv --benchmark benchmark_queries.csv

# CLIP-only baseline (local/Colab, CPU)
python build_clip_baseline_pool.py --judgements judgements.csv \
    --qdrant-url ... --qdrant-api-key ... --out clip_pool.csv
python helper-scripts/compute_metrics.py --judgements judgements.csv --pool clip_pool.csv \
  --benchmark benchmark_queries.csv --out per_query_metrics_clip_baseline.csv

# Latency benchmark (Modal, GPU)
modal run latency_benchmark.py
modal volume get abo-images latency_summary.csv .

---