USAS Multilingual Word Sense Disambiguation Datasets

This contains the Multilingual Word Sense Disambiguation (WSD) evaluation datasets, using the UCREL Semantic Analysis System (USAS) sense inventory, from the paper; Creating a Hybrid Rule and Neural Network Based Semantic Tagger using Silver Standard Data: the PyMUSAS framework for Multilingual Semantic Annotation. These evaluation datasets are a collection of existing datasets, apart from the Chinese datasets which was released with the Creating a Hybrid Rule and Neural Network Based Semantic Tagger using Silver Standard Data: the PyMUSAS framework for Multilingual Semantic Annotation paper.

All the evaluation data has either been manually tagged or has been manually checked.

For more information about the sense inventory please read the USAS guide..

Table of contents

• Uses
• Dataset Description
  • English and Finnish
  • Welsh
  • Chinese
• Dataset Statistics
• Dataset Structure
  • English
  • Finnish
  • Welsh
  • Chinese
• Citation
• License
• Dataset Card Authors
• Dataset Card Contact

Uses

These dataset can be used to evaluate Word Sense Disambiguation (WSD) models on the USAS sense inventory.

Dataset Description

We describe each dataset that is contained within the evaluation dataset in their own section;

English and Finnish

This data has come from Porting an English semantic tagger to the Finnish language by Löfberg et al. 2003, a tagged corpus of texts from a Finnish coffee website http://www.kahvilasi.net/ (this website is no longer available), the English corpus is a machine translated version of the Finnish that was post edited by a native Finnish speaker.

Welsh

It is the dataset released in Leveraging Pre-Trained Embeddings for Welsh Taggers by Ezeani et al. 2019. The dataset consists of 8 extracts from 4 diverse data sources;

• Kynulliad3 - Welsh Assembly proceedings.
• Meddalwedd - Translations of software instructions.
• Kwici - Welsh Wikipedia articles.
• LERBIML - Multi-domain spoken corpora.

Chinese

The Chinese dataset is a manually tagged text from the News Report genre of the ToRCH2019 corpus, specifically about the 2019 military world games in Wuhan, China. This dataset was manually annotated following a three-stage procedure; 1. independent tagging, 2. independent reviews of their own tagging, and 3. reaching consensus between two trained researchers.

Dataset Statistics

Dataset statistics per language:

• ISO 639-3 code - 3 digit language code.
• Text Level - How the text has been broken up.
• Texts - Number of texts
• Tokens - Number of tokens
• L. Tokens - Number of Labelled Tokens (L. Tokens)
• Multi Tag Membership - Number of labelled tokens, and the percentage (%), whereby the USAS tag either has dual, triple, or quadruple membership

An example few tokens from the Finnish test data that contains tokens with Multi Tag Membership:

Useimmat_N5+++ kaupan_I2.2/H1 kahvipaketit_O2/F2 ovat_A3+ useiden_N5+ eri_A6.1- kahvilajikkeiden_A4.1/F2

The tokens kaupan, kahvipaketit, and kahvilajikkeiden all contain dual tag membership, in the kaupan case the dual membership is I2.2 and H1.

These dataset statistics were generated after the data was pre-processed; removed all tag tokens marked as punctuation or containing the unmatched USAS tag/label ("Z99"), as these tags do not have any semantic meaning. In addition, any labelled tokens that could not be matched to the USAS tagset were removed, in addition if the token contained more than one USAS label assigned to them we used the first label (this affected 6 tokens in the Chinese dataset, this multi label token assignment is not the same as Multi tag membership as multi tag membership still counts as one USAS label).

Dataset Structure

The dataset structure varies by file, here we detail the different data structures.

English

Each new line represents a sentence of annotated tokens. Each annotated token is split by whitespace. Each annotated token is represented like the following {Token Text}_{USAS TAG}{[i\d+.\d+.\d+}:

Example annotated tokens:

Vac_F2/O2[i136.2.1 pot_F2/O2[i136.2.2 is_A3+

• Token Text - The text representing the annotated token.
• USAS Tag - The USAS tag/label.
• [i\d+.\d+.\d+ - A special sequence indicating the token is part of a Multi Word Expression (MWE). The first set of digits represents a unique ID, the second set represents the number of tokens in the MWE, and the last set represents the token index in the MWE.

Finnish

Each new line represents a sentence of annotated tokens. Each annotated token is split by whitespace. Each annotated token is represented like the following {Token Text}_{USAS TAG}_{i}?:

Example annotated tokens:

Vac_F2/O2_i pot_F2/O2_i on_A3+

• Token Text - The text representing the annotated token.
• USAS Tag - The USAS tag/label.
• i - if present states that it is part of a Multi Word Expression.

Welsh

Each new line represents a sentence of annotated tokens. Each annotated token is split by whitespace. Each annotated token is represented like the following whereby each piece of information is segmented by the pipe character | {Token}|{Lemma}|{Core POS}|{True CorCenCC Basic POS}|{Predicted CorCenCC Enriched POS}|{Predicted CorCenCC Basic POS}|{USAS Tag}:

Example annotated token:

A|a|pron|Rha|Rhaperth|Rha|Z5

• Token - The annotated token.
• Lemma - The predicted lemma. The prediction was made from the CyTag tagger.
• Core POS - Mapping from the True CorCenCC Basic POS tag to the Core POS tag. This mapping is based off table A.1 in Leveraging Pre-Trained Embeddings for Welsh Taggers.
• True CorCenCC Basic POS - Human annotated CorCenCC Basic POS tag.
• Predicted CorCenCC Enriched POS - this has come from running the CyTag tagger. As this tag has been predicted it may be different to the True CorCenCC Basic POS.
• Predicted CorCenCC Basic POS - this has come from running the CyTag tagger. As this tag has been predicted it may be different to the True CorCenCC Basic POS.
• USAS Tag - The USAS tag/label.

For more details on how this dataset was generated please see the following GitHub repository.

Chinese

The data is stored in a CSV file with the following headings:

Token,POS,Predicted-USAS,Corrected-USAS,Errors,Notes,sentence-break

• Token - the token text
• POS - the predicted token's Part Of Speech whereby it uses the Universal Dependency POS tagset
• Predicted-USAS - The semicolon list of predicted USAS tags, whereby the first USAS tag in the list should be the most probable, e.g. "H4;I1.2"
• Corrected-USAS - The semicolon list of corrected USAS tags, whereby the first USAS tag in the list should be the most probable, e.g. "H1;I1.1"
• Errors - A semicolon list of errors that the annotators have added when annotating. Notes - Any additional notes the annotators have added while annotating.
• sentence-break - False or True. If True that token is the last token in the sentence.

To generate the tokens, POS, and predicted USAS tags we used spaCy zh_core_web_trf version 3.8 model and PyMUSAS version 0.3.0 with cmn_dual_upos2usas_contextual-0.3.3 as the USAS semantic tagger.

Citation

Each dataset should be cited separately, but for reference all 4 datasets were used as evaluation datasets within Creating a Hybrid Rule and Neural Network Based Semantic Tagger using Silver Standard Data: the PyMUSAS framework for Multilingual Semantic Annotation paper.

• English and Finnish - Porting an English semantic tagger to the Finnish language by Löfberg et al. 2003
• Welsh - Leveraging Pre-Trained Embeddings for Welsh Taggers by Ezeani et al. 2019
• Chinese - Creating a Hybrid Rule and Neural Network Based Semantic Tagger using Silver Standard Data: the PyMUSAS framework for Multilingual Semantic Annotation.

License

CC-BY-NC-SA 4.0

Dataset Card Authors

• UCREL (ucrel@lancaster.ac.uk)
• Andrew Moore / apmoore1 (a.p.moore@lancaster.ac.uk / andrew.p.moore94@gmail.com)
• Paul Rayson (p.rayson@lancaster.ac.uk)

Dataset Card Contact

• UCREL (ucrel@lancaster.ac.uk)
• Andrew Moore / apmoore1 (a.p.moore@lancaster.ac.uk / andrew.p.moore94@gmail.com)
• Paul Rayson (p.rayson@lancaster.ac.uk)