WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Overview

We introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM’s internal knowledge when retrieval is unreliable.

🎯 Todo List

• Release paper on Arxiv.
• Publish the details of dataset processing.
• Release the multi-modal retrieval code along with the corresponding knowledge base.
• Release the RL training code for Refiner.
• Release the Setup details.

Setup

conda create -n wikis python=3.10
pip install -r requirements.txt

# faiss
conda install faiss==1.7.4

# flash attention 2
pip install flash-attn --no-build-isolation

pip install -e .

Data Preparation

You can refer to the EchoSight VQA Questions section for preparing the EVQA and InfoSeek datasets. However, note the following important points:

1. When downloading the iNaturalist 2021 images, ensure you download the train set (224GB) and val set (8.4GB) for training and testing purposes, respectively.
2. Following the OMGM procedure, all images from the iNaturalist train and val sets have been reconstructed and converted to the id.jpg format, all stored in a single path (eliminating the need for id2name mapping).
3. For the Google Landmarks Dataset V2, ensure you download the train set (499 tar files), not the test set mistakenly mentioned in the EchoSight documentation.
4. As for the oven_images, decompress the 8 tar files and consolidate all images into the oven_imgs directory.

After setting up all datasets, please update the respective dataset paths in utils/utils.py.

Launch Retriever Service

You can download the necessary multi-modal kb eva_qwen3_faiss_index from our 🤗 Hugging Face to start the service.

bash scripts/start_retriever_service.sh

GRPO Training for Refiner

You can download rl training data from our 🤗 Hugging Face Repo. Then, you can start the training process using the following script:

bash scripts/run_evqa_grpo.sh

Generate Refined Queries

bash scripts/vllm_refiner.sh
bash scripts/gen_refined_queries.sh

Multi-modal Retrieval

bash scripts/run_retrieval.sh

📚 Citation

If you find our work useful, please give us a star 🌟 and use the following BibTeX entry for citation.

@article{zhu2026wikiseeker,
  title={WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering},
  author={Zhu, Yingjian and Wang, Xinming and Ding, Kun and Wang, Ying and Fan, Bin and Xiang, Shiming},
  journal={arXiv preprint arXiv:2604.05818},
  year={2026}
}

Acknowledgements

Our code is built upon EchoSight, OMGM and DeepRetrieval. Thanks for their great work.