📰 News

• 🔥 UOUO is accepted by EMNLP 2024 Main Conference!
• 📢 We have decided to release our full pipeline, including web scraping, to support and accelerate dataset research efforts within the vision-language model community! (2025.08)
• 📢 Our raw data are released!
• 📢 All the evaluation and data generation scripts are released in this public repo! (Previously, raw data and evaluation scripts were only available upon request.)

⚙️ Environment Setup

We use python 3.10 and cuda 12.1 in our setting.

1. Create Conda Environment

conda create -n uouo python=3.10
conda activate uouo

2. Install Pytorch and Tranformers packages

pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.45.0.dev0

3. Install other dependencies

pip install genai openai accelerate tqdm

📊 Data

Scraped Domain-Specific Data

We release the raw download data and the preprocessed data (link coming soon).

UOUO Benchmark

We release the mosaic images evaluated in the paper, as well as the human-cleaned images from which the mosaics were constructed.

Dataset Details

We provide direct download links for each dataset to allow users to download them directly from the internet.
In addition, we release the category names used for scraping data.

🔍 Evaluation

Note: rand stands for no-mmd setting.

See the evaluation section for more details.

🪄 Create Mosaic Images

See the mosaic-generation section for more details.

💻 Extract Compute top-5 closest categories by CLIP-MMD

See the clip-mmd section for more details.

🙏 Acknowlegement

Tran Van, H. (2023). Mosaic Data Augmentation for YOLO. GitHub. https://github.com/huytranvan2010/Mosaic_data_augmentation_YOLO
OpenAI. (2021). CLIP: Contrastive Language-Image Pre-Training. GitHub. https://github.com/openai/CLIP

📜 License

This project is released under the MIT License.

🔗 Citation

If you use our work, please cite:

@misc{pi2024uououncontextualizeduncommonobjects,
      title={UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models}, 
      author={Xinyu Pi and Mingyuan Wu and Jize Jiang and Haozhen Zheng and Beitong Tian and Chengxiang Zhai and Klara Nahrstedt and Zhiting Hu},
      year={2024},
      eprint={2407.18391},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.18391}, 
}