Multi³Hate Dataset: Multimodal, Multilingual, and Multicultural Hate Speech Dataset

Warning: this dataset contains content that may be offensive or upsetting

Multi³Hate dataset, introduced in our paper: Multi³Hate: Advancing Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models

Abstract: Hate speech moderation on global platforms poses unique challenges due to the multimodal and multilingual nature of content, along with the varying cultural perceptions. How well do current vision-language models (VLMs) navigate these nuances? To investigate this, we create the first multimodal and multilingual parallel hate speech dataset, annotated by a multicultural set of annotators, called Multi3Hate. It contains 300 parallel meme samples across 5 languages: English, German, Spanish, Hindi, and Mandarin. We demonstrate that cultural background significantly affects multimodal hate speech annotation in our dataset. The average pairwise agreement among countries is just 74%, significantly lower than that of randomly selected annotator groups. Our qualitative analysis indicates that the lowest pairwise label agreement—only 67% between the USA and India—can be attributed to cultural factors. We then conduct experiments with 5 large VLMs in a zero-shot setting, finding that these models align more closely with annotations from the US than with those from other cultures, even when the memes and prompts are presented in the dominant language of the other culture.

Dataset Details

The dataset comprises a curated collection of 300 memes—images paired with embedded captions—a prevalent form of multimodal content, presented in five languages: English (en), German (de), Spanish (es), Hindi (hi), and Mandarin (zh). Each of the 1,500 memes (300×5 languages) is annotated for hate speech in the respective target language by at least five native speakers from the same country. These countries were chosen based on the largest number of native speakers of each target language: USA (US), Germany (DE), Mexico (MX), India (IN), and China (CN).

We curate a list of culturally relevant templates by filtering them according to sociopolitical categories. These categories were discussed and decided among the authors. Each category is further divided into specific topics based on established criteria.

Dataset Sources

• Repository: Link
• Paper: Link

Annotation process & Demographic

We refer to our paper for more details.

Who are the source data producers?

The original meme images were crawled from https://memegenerator.net

License

CC BY-NC-ND 4.0

Citation

BibTeX:

@misc{bui2024multi3hatemultimodalmultilingualmulticultural,
      title={Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models}, 
      author={Minh Duc Bui and Katharina von der Wense and Anne Lauscher},
      year={2024},
      eprint={2411.03888},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.03888}, 
}