Factual Visual Question Answering (FVQA)

Dataset Summary

FactualVQA (FVQA) is a multimodal Visual Question Answering dataset created for search-augmented training and evaluation. It emphasizes knowledge-intensive questions that require external information beyond the given image. Each entry includes an image, a question, and an answer (optionally accompanied by candidate answers), enabling models to develop and refine on-demand search strategies. Details of dataset construction are provided in the the blog or the paper.

Dataset Structure

• Data Fields The datasets are stored in Parquet format and include the following columns:
  • data_id: unique data id
  • prompt: The user question
  • images: Raw image data stored as bytes
  • reward_model: Ground truth and candidate answers used for reward calculation
  • data_source: Specifies which reward function to use in veRL (e.g., mmsearch_r1/fvqa_train, mmsearch_r1/fvqa_test)
  • image_urls: Optional field for potential use with the image search tool
  • category: search-required and search-free annotations
• Train/Test Split
  • Train: ~5k samples, with approximately 68% search-required and 32% search-free (estimated using Qwen2.5-VL-7B-Instruct)
  • Test: ~1.8k samples
• Source
  • Image Sources: Google Image Search, subset of InfoSeek's training split
  • QA Sources: GPT4o-generated, Human-annotated(for test split), subset of InfoSeek's training split
• Cached Image Search Results of FVQA
  • Cached Image Search Results (relevant webpage titles and thumbnail-image-urls) of images of FVQA dataset, indexed by data_id
    • fvqa_train_image_search_results_cache.pkl
    • fvqa_test_image_search_results_cache.pkl
  • Since the webpage thumbnail URLs from SerpAPI’s search results include both strings and PIL.Image objects (e.g., <class 'PIL.JpegImagePlugin.JpegImageFile'>), you may need to pip3 install pillow==11.1.0 to load the pickle files

Citation

@article{wu2025mmsearch,
  title={MMSearch-R1: Incentivizing LMMs to Search},
  author={Wu, Jinming and Deng, Zihao and Li, Wei and Liu, Yiding and You, Bo and Li, Bo and Ma, Zejun and Liu, Ziwei},
  journal={arXiv preprint arXiv:2506.20670},
  year={2025}
}