🎨 ArtSeek

Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

Nicola Fanelli · Gennaro Vessio · Giovanna Castellano
University of Bari Aldo Moro

ArtSeek answers open-ended questions about artworks. Given an image and a question, a multimodal LLM looks at the painting, reads an artwork card predicted by a classifier, decides when it needs external knowledge, and retrieves it from Wikipedia before answering. The language model is used zero-shot, and the repository includes an interactive demo.

✨ Highlights

• WikiFragments, a multimodal knowledge base: 8.2M fragments (a paragraph plus the images above it, rendered as one image) from 973k visual-arts pages of English Wikipedia, searched with ColQwen2 late interaction.
• Two-stage retrieval that keeps only the text tokens of a multimodal query and prefetches on pooled vectors, matching full multi-vector retrieval at under 7% of its cost.
• LICN, a late-interaction classifier for artist, genre, style, media and tags that beats prior work on ArtGraph and works over an open label set.
• In-context tool calling: a single worked example teaches Qwen2.5-VL-32B when and what to retrieve, over several steps.

🔍 How it works

Multimodal retrieval. The image and the question are encoded together by ColQwen2; only the question tokens are kept, candidates are prefetched on pooled fragment vectors and reranked on the full ones.

Late Interaction Classification Network. Frozen ColQwen2 embeddings of the image and of every label, concatenated with learnable task embeddings, are encoded by a shared transformer and matched per task. The predictions form the artwork card.

In-context reasoning. The one-shot example shown to the model: visual questions are answered directly, knowledge questions trigger get_relevant_documents calls whose fragments are read before answering.

📊 Results at a glance

Question answering, mean correctness on a 0-2 scale judged by phi-4, with 95% confidence intervals (paper, Tab. 3). ArtSeek gains where its knowledge base covers the painting and ties with its backbone where it does not.

The same gain holds with Gemma-3-27B and Mistral-Small-3.1-24B as backbones (+0.22 to +0.28 on ArtPedia-VQA). LICN reaches 71.8 / 78.5 / 69.8 top-1 on ArtGraph artist / genre / style. Captioning, retrieval and classification tables, with the commands to reproduce them, are in data/README.md.

🖼️ Examples

The question names nothing to search for; the image and the card turn it into a query, and a retrieved fragment corrects the backbone's “hennin” into the right balzo (even though the card's artist is wrong).

ArtSeek compared with Qwen2.5-VL and ChatGPT. Right: a failure case, where retrieval suggests a plausible but wrong battle.

📑 Contents

1. Requirements
2. Installation
3. Download the resources
4. Build the vector store
5. Run the demo
6. Use ArtSeek from Python
7. Reproduce the paper's evaluation
8. Rebuild the resources from scratch

The additional experiments of the revised paper (question-answering benchmarks, other backbones, human evaluation) are on the rebuttal branch.

1. Requirements

Setting everything up takes about a day, mostly unattended: downloading WikiFragments, ingesting it into Qdrant (~2.5 h with 4 processes) and building the index (~8 h).

2. Installation

git clone https://github.com/cilabuniba/artseek.git
cd artseek
uv sync                  # creates .venv with PyTorch 2.8 (CUDA 12.6) and vLLM 0.11
source .venv/bin/activate
cp .env.example .env     # then edit it

.env is read by the demo and the scripts. The most important settings:

The default generation engine is vLLM. The transformers engine (ARTSEEK_ENGINE=hf) is slower and needs FlashAttention: uv pip install flash-attn --no-build-isolation.

3. Download the resources

Everything is hosted on the Hugging Face Hub:

python scripts/download_resources.py                 # everything (several hours)
python scripts/download_resources.py --skip-dataset  # models only

The script downloads into $HF_HOME and prepares WikiFragments with datasets.load_dataset, which is how ArtSeek loads it. After it finishes you can set HF_HUB_OFFLINE=1 and HF_DATASETS_OFFLINE=1 in .env.

4. Build the vector store

ArtSeek queries a Qdrant collection named wikifragments-visual-arts-embeds over gRPC (port 6334). We used Qdrant v1.17.0.

4.1 Start Qdrant

With Docker (workstations):

docker run -d --name qdrant -p 6333:6333 -p 6334:6334 \
    -v /path/to/qdrant_storage:/qdrant/storage qdrant/qdrant:v1.17.0

Without Docker (e.g. HPC clusters): download a Linux binary from the Qdrant releases, or build it from source (below). Then:

export QDRANT_BIN=/path/to/qdrant QDRANT_STORAGE=/path/to/qdrant_storage
bash scripts/start_qdrant.sh    # returns when the server is ready

# Protobuf compiler
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v25.1/protoc-25.1-linux-x86_64.zip
unzip -o protoc-25.1-linux-x86_64.zip -d proto_bin
export PROTOC=$PWD/proto_bin/bin/protoc PATH=$PWD/proto_bin/bin:$PATH
# If Clang is not found, point LIBCLANG_PATH to the lib/ directory of your LLVM install.

git clone https://github.com/qdrant/qdrant.git
cd qdrant && git checkout v1.17.0
RUSTFLAGS="-C target-cpu=native" cargo build --release --bin qdrant
# binary: target/release/qdrant

4.2 Ingest the embeddings (~2.5 h with 4 processes)

With Qdrant running and the dataset downloaded:

python -m artseek.data.main make-qdrant-store --process-idx 0 --num-proc 1

To go faster, run N processes in parallel with --process-idx 0 … N-1 and --num-proc N. Process 0 (re)creates the collection; the others wait 5 minutes before uploading. slurm/build_qdrant_store.sh does this with 4 processes.

4.3 Build the index (~8 h)

python -m artseek.data.main add-qdrant-index

The command polls the collection every 5 minutes and returns when indexing is complete (slurm/add_qdrant_index.sh). Check the result:

curl -s localhost:6333/collections/wikifragments-visual-arts-embeds | python -m json.tool | grep points_count
# "points_count": 8166323

5. Run the demo

With Qdrant running and the resources downloaded:

streamlit run app.py

Open http://localhost:8501 and upload an image. The first upload loads the models, which takes a few minutes; LICN then shows the artwork card. Ask questions in the chat: the model decides when to retrieve, and each retrieval shows the query and the retrieved fragments. The reasoning (<think> block) and a timing breakdown are expandable. The sidebar toggles classification and retrieval, so you can compare with the bare backbone.

On a remote GPU node, start Streamlit there and forward the port:

# on the GPU node
streamlit run app.py --server.port 8501 --server.address 0.0.0.0 --server.headless true
# on your machine
ssh -N -L 8501:<gpu-node>:8501 <user>@<login-node>

slurm/demo.sh starts Qdrant and the demo in one SLURM job and prints the SSH command. Fill in the #SBATCH placeholders first.

If vLLM runs out of memory during CUDA-graph capture, or the retriever runs out of memory, lower ARTSEEK_GPU_MEMORY_UTILIZATION (or ARTSEEK_MAX_MODEL_LEN).

6. Use ArtSeek from Python

from langchain_core.messages import HumanMessage
from PIL import Image

from artseek.method.generate.config import load_artseek
from artseek.method.generate.prompts import build_user_content

model = load_artseek()  # vLLM engine, default configuration

image = Image.open("painting.jpg").convert("RGB")
card = model.classify(image)  # {"artist": [("claude-monet", 0.87)], "genre": ..., ...}

messages = [HumanMessage(content=build_user_content("Describe the historical context of this painting.", card))]
new_messages, context_images = model.chat_turn(messages, image, classify=True, retrieve=True)
print(new_messages[-1].content)

# Follow-up question: keep the history and the retrieved images.
messages += new_messages + [HumanMessage(content="Who commissioned it?")]
more, more_images = model.chat_turn(messages, image, context_images=context_images)

• Without classification, pass card=None and classify=False. With retrieve=False the model answers from its own knowledge.
• new_messages contains the tool calls and ToolMessages. Their artifact holds the retrieved documents (documents, context_idxs), and the final message's response_metadata["step_durations"] holds the timing.
• load_artseek(**overrides) accepts any Qwen2_5_VLRAGModel argument, e.g. model_kwargs for vLLM or use_shot=False to describe the tool-calling policy in the system prompt instead of the one-shot example.
• Other backbones: gemma3_vllm.py and mistral3_vllm.py wrap Gemma 3 and Mistral Small 3.1 (pass them as chat_model_cls). They cannot emit Qwen-style tool calls, so use them with seeded retrieval: chat_turn(..., seed_queries=[{"query": ..., "requires_image": True}], max_tool_calls=0).

Code layout:

artseek/
  method/
    generate/    ArtSeek pipeline: rag.py (model + chat_turn), config.py (defaults),
                 prompts.py, tools.py, graph.py (LangGraph batch inference),
                 vLLM / transformers backbones, evaluation (test.py, eval.py)
    retrieve/    ColQwen2 + Qdrant retriever, retrieval evaluation
    classify/    LICN model and training
  data/          data pipeline (ArtGraph, Wikipedia, WikiFragments, Qdrant) and dataset loaders
app.py           Streamlit demo
scripts/         resource download, Qdrant launcher
slurm/           SLURM templates (placeholders for account/partition)
models/configs/  experiment configurations

7. Reproduce the paper's evaluation

data/README.md explains how to download and lay out every dataset of the paper and how to run each experiment, with the results to expect:

8. Rebuild the resources from scratch

Not needed to use ArtSeek: everything above is on the Hugging Face Hub. These are the steps that produced it.

ArtGraph (Neo4j), for LICN training data

1. Install Neo4j Community 4.4.47 and the APOC plugin 4.4.0.24 (in plugins/, enabled in conf/neo4j.conf).
2. Load the ArtGraph dump: bin/neo4j-admin load --from=artgraph2.0.dump --database=neo4j --force.
3. Put the connection details in .env (NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD).

Data pipeline

All steps are subcommands of python -m artseek.data.main, and they write to data/ (or $ARTSEEK_DATA_DIR):

Step 5 runs a modified WikiExtractor on the English Wikipedia dump (enwiki-latest-pages-articles.xml.bz2, 2024-09-20). The modification keeps image URLs and captions as <a> tags:

python WikiExtractorNew.py --json -s --lists --links enwiki-latest-pages-articles.xml.bz2 -o data/texts/text_en

scripts/fetch_wikifragments_licenses.py collects the license and attribution of every image.

LICN training

accelerate launch -m artseek.method.classify.train_li_classification_network train \
    --config-path models/configs/classify/li_classification_network_tft.yaml

tft is the configuration used in the paper; the other files are the ablations.

Citation

@article{fanelli2025artseek,
  title   = {ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval},
  author  = {Fanelli, Nicola and Vessio, Gennaro and Castellano, Giovanna},
  journal = {arXiv preprint arXiv:2507.21917},
  year    = {2025}
}

License

Code: MIT. WikiFragments text is CC BY-SA 4.0 (Wikipedia); each image keeps its own license, listed in the dataset.