Scaling for Cultural Equity: Investigating the Impact of Training Data Size on Cultural Bias in Vision-Language Models

Final project repository for DS-GA 1012 Natural Language Understanding and Computational Semantics by Prof Tal Linzen in Spring 2025.

Authors

Vy Mai (nm4867@nyu.edu), Eric Zhao (ez2545@nyu.edu). Emily Wang (eyw2010@nyu.edu), Kristina Fujimoto (kef7529@nyu.edu)

Overview

Methodology

1. Models

2. Datasets

2.1. LAION

2.2. Wikipedia image-caption evaluation dataset

3. Experimental Setup

3.1. Fine-tune open-source VLMs on LAION training datasets of varying size

3.2 Curate benchmark of cultural images and corresponding captions by web scraping Wikipedia

3.3. Evaluate model performance with cultural questions and answers

3.4. Analyze and compare results across models to assess their performance on culturally nuanced concepts.

Results