E-commerce Multimodal Visual Search

A production-grade multimodal visual search system for cross-border e-commerce products. Search a large product catalog by image, text, or image + text — powered by state-of-the-art multimodal embeddings and vector retrieval.

Architecture

graph LR
    A[Gradio UI<br/>Upload / Text] -->|query| B[Search Engine]
    B -->|encode| C[Visualized_BGE<br/>Multimodal Embeddings]
    C -->|vector| B
    B -->|ANN search| D[Milvus Lite<br/>Vector Database]
    D -->|top-K results| B
    B -->|results| A

    E[Amazon Products<br/>Dataset] -->|offline indexing| F[Batch Encoder]
    F -->|embeddings + metadata| D

Data Flow

1. Offline Indexing: Download Amazon product images → Encode with Visualized_BGE (image + product name) → Insert embeddings into Milvus Lite
2. Online Search: User uploads image / enters text → Encode query → Vector similarity search in Milvus → Return top-K results with metadata

Features

• Three Search Modes:
  • Image Search: Upload a product image to find visually similar items
  • Text Search: Describe what you're looking for in natural language
  • Multimodal Search: Combine image + text for precision retrieval (e.g., "like this but in blue")
• Multimodal Embeddings: Visualized_BGE from BAAI — jointly encodes images and text into a shared 768-dim vector space
• Vector Database: Milvus Lite for fast approximate nearest neighbor search, no server required
• Gradio Demo: Clean 3-tab web interface with gallery view, similarity scores, and product metadata
• Production Patterns: Singleton encoder, batch indexing, concurrent image downloads, graceful error handling

Tech Stack

Quick Start

Prerequisites

• Python 3.10+
• uv package manager (recommended) or pip
• ~2GB disk space for model weights + images
• NVIDIA GPU recommended (CPU works but slower)

1. Install Dependencies

# Clone the repo
git clone <repo-url>
cd ecommerce-visual-search

# Using uv (recommended)
uv sync

# Or using pip
pip install -r requirements.txt

2. Download Model & Data

# Download Visualized_BGE model weights (~350MB)
uv run python models/download_model.py

# Download product dataset + images (default: 5000 products)
uv run python data/download_dataset.py --limit 1000

3. Build Index

# Encode all products and build vector index
uv run python indexing/build_index.py

4. Launch Demo

# Start Gradio web interface
uv run python app/gradio_app.py
# Open http://localhost:7860 in your browser

Project Structure

ecommerce-visual-search/
├── config.py                    # Centralized configuration (paths, model, Milvus, constants)
├── requirements.txt             # Python dependencies
│
├── data/
│   ├── download_dataset.py      # HuggingFace dataset download + batch image download
│   └── images/                  # Downloaded product images (gitignored)
│
├── models/
│   ├── download_model.py        # Download Visualized_BGE weights from HuggingFace
│   └── weights/                 # Model weights (gitignored)
│
├── indexing/
│   ├── encoder.py               # MultimodalEncoder: encode(image, text) → 768-dim vector
│   └── build_index.py           # Batch encode + insert into Milvus with metadata
│
├── search/
│   └── search_engine.py         # SearchEngine: image_search / text_search / multimodal_search
│
├── app/
│   └── gradio_app.py            # Gradio demo: 3 tabs with gallery + similarity scores
│
├── scripts/
│   └── evaluate.py              # Retrieval evaluation: Recall@1, @5, @10
│
└── .gitignore

Search Modes Explained

Image Search (以图搜图)

Upload a product photo → The system encodes it using Visualized_BGE's image encoder → Finds the most visually similar products in the database via cosine similarity.

Text Search (文字搜图)

Enter a text description like "wireless bluetooth headphones" → Encoded into the same vector space as product images → Retrieves semantically matching products.

Multimodal Search (图文联合搜索)

Combine an image with a text modifier. For example:

• Upload a photo of red shoes + type "but in blue" → Finds blue shoes with similar style
• Upload a laptop image + type "gaming laptop with RGB" → Finds gaming variants

This works because Visualized_BGE fuses image and text features into a single embedding.

Configuration

All configuration is centralized in config.py:

License

MIT