Dataset Description

• Homepage: https://semantic-textual-relatedness.github.io
• Repository: GitHub
• Paper: SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 14 Languages
• Paper: SemEval Task 1: Semantic Textual Relatedness for African and Asian Languages
• Leaderboard: https://codalab.lisn.upsaclay.fr/competitions/16799#results
• Point of Contact: Nedjma Ousidhoum

Dataset Summary

SemRel2024 is a collection of Semantic Textual Relatedness (STR) datasets for 14 languages, including African and Asian languages. The datasets are composed of sentence pairs, each assigned a relatedness score between 0 (completely) unrelated and 1 (maximally related) with a large range of expected relatedness values. SemRel2024 dataset was used as part of the SemEval2024 shared task 1. The task aims to evaluate the ability of systems to measure the semantic relatedness between two sentences.

Languages

The SemRel2024 dataset covers the following 14 languages:

1. Afrikaans (afr)
2. Algerian Arabic (arq)
3. Amharic (amh)
4. English (eng)
5. Hausa (hau)
6. Indonesian (ind)
7. Hindi (hin)
8. Kinyarwanda (kin)
9. Marathi (mar)
10. Modern Standard Arabic (arb)
11. Moroccan Arabic (ary)
12. Punjabi (pan)
13. Spanish (esp)
14. Telugu (tel)

Note: Spanish test labels are all -1 because the Spanish team retained the gold test labels to avoid contamination problems in future benchmarking. We refer to the CodaLab contest website to evaluate your predictions, which will remain open.

Dataset Structure

Data Instances

Each instance in the dataset consists of two text segments and a relatedness score indicating the degree of semantic relatedness between them.

{
  "sentence1": "string",
  "sentence2": "string",
  "label": float
}

• sentence1: a string feature representing the first text segment.
• sentence2: a string feature representing the second text segment.
• label: a float value representing the semantic relatedness score between sentence1 and sentence2, typically ranging from 0 (not related at all) to 1 (highly related).

Citation Information

If you use the SemRel2024 dataset in your research, please cite the following papers:

@misc{ousidhoum2024semrel2024,
title={SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 14 Languages}, 
author={Nedjma Ousidhoum and Shamsuddeen Hassan Muhammad and Mohamed Abdalla and Idris Abdulmumin and Ibrahim Said Ahmad and
Sanchit Ahuja and Alham Fikri Aji and Vladimir Araujo and Abinew Ali Ayele and Pavan Baswani and Meriem Beloucif and
Chris Biemann and Sofia Bourhim and Christine De Kock and Genet Shanko Dekebo and
Oumaima Hourrane and Gopichand Kanumolu and Lokesh Madasu and Samuel Rutunda and Manish Shrivastava and
Thamar Solorio and Nirmal Surange and Hailegnaw Getaneh Tilaye and Krishnapriya Vishnubhotla and Genta Winata and
Seid Muhie Yimam and Saif M. Mohammad},
      year={2024},
      eprint={2402.08638},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@inproceedings{ousidhoum-etal-2024-semeval, 
title = "{S}em{E}val-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages",
author = "Ousidhoum, Nedjma and Muhammad, Shamsuddeen Hassan and Abdalla, Mohamed and Abdulmumin, Idris and
Ahmad,Ibrahim Said and Ahuja, Sanchit and Aji, Alham Fikri and Araujo, Vladimir and     Beloucif, Meriem and
De Kock, Christine and Hourrane, Oumaima and Shrivastava, Manish and Solorio, Thamar and Surange, Nirmal and
Vishnubhotla, Krishnapriya and Yimam, Seid Muhie and Mohammad, Saif M.",
booktitle = "Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)",
year = "2024",
publisher = "Association for Computational Linguistics"
}