Dataset Card for JSICK

Table of Contents

• Dataset Card for JSICK
  • Table of Contents
  • Dataset Description
    • Dataset Summary
      • Japanese Sentences Involving Compositional Knowledge (JSICK) Dataset.
      • JSICK-stress Test set
    • Languages
  • Dataset Structure
    • Data Instances
      • base
      • stress
    • Data Fields
      • base
      • stress
    • Data Splits
    • Annotations
  • Additional Information
    • Licensing Information
    • Citation Information
    • Contributions

Dataset Description

• Homepage: https://github.com/verypluming/JSICK
• Repository: https://github.com/verypluming/JSICK
• Paper: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00518/113850/Compositional-Evaluation-on-Japanese-Textual
• Paper: https://www.jstage.jst.go.jp/article/pjsai/JSAI2021/0/JSAI2021_4J3GS6f02/_pdf/-char/ja

Dataset Summary

From official GitHub:

Japanese Sentences Involving Compositional Knowledge (JSICK) Dataset.

JSICK is the Japanese NLI and STS dataset by manually translating the English dataset SICK (Marelli et al., 2014) into Japanese. We hope that our dataset will be useful in research for realizing more advanced models that are capable of appropriately performing multilingual compositional inference.

JSICK-stress Test set

The JSICK-stress test set is a dataset to investigate whether models capture word order and case particles in Japanese. The JSICK-stress test set is provided by transforming syntactic structures of sentence pairs in JSICK, where we analyze whether models are attentive to word order and case particles to predict entailment labels and similarity scores.

The JSICK test set contains 1666, 797, and 1006 sentence pairs (A, B) whose premise sentences A (the column sentence_A_Ja_origin) include the basic word order involving ga-o (nominative-accusative), ga-ni (nominative-dative), and ga-de (nominative-instrumental/locative) relations, respectively.

We provide the JSICK-stress test set by transforming syntactic structures of these pairs by the following three ways:

• scrum_ga_o: a scrambled pair, where the word order of premise sentences A is scrambled into o-ga, ni-ga, and de-ga order, respectively.
• ex_ga_o: a rephrased pair, where the only case particles (ga, o, ni, de) in the premise A are swapped
• del_ga_o: a rephrased pair, where the only case particles (ga, o, ni) in the premise A are deleted

Languages

The language data in JSICK is in Japanese and English.

Dataset Structure

Data Instances

When loading a specific configuration, users has to append a version dependent suffix:

import datasets as ds

dataset: ds.DatasetDict = ds.load_dataset("hpprc/jsick")
print(dataset)
# DatasetDict({
#     train: Dataset({
#         features: ['id', 'premise', 'hypothesis', 'label', 'score', 'premise_en', 'hypothesis_en', 'label_en', 'score_en', 'corr_entailment_labelAB_En', 'corr_entailment_labelBA_En', 'image_ID', 'original_caption', 'semtag_short', 'semtag_long'],
#         num_rows: 4500
#     })
#     test: Dataset({
#         features: ['id', 'premise', 'hypothesis', 'label', 'score', 'premise_en', 'hypothesis_en', 'label_en', 'score_en', 'corr_entailment_labelAB_En', 'corr_entailment_labelBA_En', 'image_ID', 'original_caption', 'semtag_short', 'semtag_long'],
#         num_rows: 4927
#     })
# })

dataset: ds.DatasetDict = ds.load_dataset("hpprc/jsick", name="stress")
print(dataset)
# DatasetDict({
#     test: Dataset({
#         features: ['id', 'premise', 'hypothesis', 'label', 'score', 'sentence_A_Ja_origin', 'entailment_label_origin', 'relatedness_score_Ja_origin', 'rephrase_type', 'case_particles'],
#         num_rows: 900
#     })
# })

base

An example of looks as follows:

{
    'id': 1,
    'premise': '子供たちのグループが庭で遊んでいて、後ろの方には年を取った男性が立っている',
    'hypothesis': '庭にいる男の子たちのグループが遊んでいて、男性が後ろの方に立っている',
    'label': 1, // (neutral)
    'score': 3.700000047683716,
    'premise_en': 'A group of kids is playing in a yard and an old man is standing in the background',
    'hypothesis_en': 'A group of boys in a yard is playing and a man is standing in the background',
    'label_en': 1, // (neutral)
    'score_en': 4.5,
    'corr_entailment_labelAB_En': 'nan',
    'corr_entailment_labelBA_En': 'nan',
    'image_ID': '3155657768_b83a7831e5.jpg',
    'original_caption': 'A group of children playing in a yard , a man in the background .',
    'semtag_short': 'nan',
    'semtag_long': 'nan',
}

stress

An example of looks as follows:

{
    'id': '5818_de_d',
    'premise': '女性火の近くダンスをしている',
    'hypothesis': '火の近くでダンスをしている女性は一人もいない',
    'label': 2,  // (contradiction)
    'score': 4.0,
    'sentence_A_Ja_origin': '女性が火の近くでダンスをしている',
    'entailment_label_origin': 2,
    'relatedness_score_Ja_origin': 3.700000047683716,
    'rephrase_type': 'd',
    'case_particles': 'de'
}

Data Fields

base

A version adopting the column names of a typical NLI dataset.

stress

Data Splits

Annotations

To annotate the JSICK dataset, they used the crowdsourcing platform "Lancers" to re-annotate entailment labels and similarity scores for JSICK. They had six native Japanese speakers as annotators, who were randomly selected from the platform. The annotators were asked to fully understand the guidelines and provide the same labels as gold labels for ten test questions.

For entailment labels, they adopted annotations that were agreed upon by a majority vote as gold labels and checked whether the majority judgment vote was semantically valid for each example. For similarity scores, they used the average of the annotation results as gold scores. The raw annotations with the JSICK dataset are publicly available. The average annotation time was 1 minute per pair, and Krippendorff's alpha for the entailment labels was 0.65.

Additional Information

• verypluming/JSICK
• Compositional Evaluation on Japanese Textual Entailment and Similarity
• JSICK: 日本語構成的推論・類似度データセットの構築

Licensing Information

CC BY-SA 4.0

Citation Information

@article{yanaka-mineshima-2022-compositional,
    title = "Compositional Evaluation on {J}apanese Textual Entailment and Similarity",
    author = "Yanaka, Hitomi  and
      Mineshima, Koji",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "10",
    year = "2022",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2022.tacl-1.73",
    doi = "10.1162/tacl_a_00518",
    pages = "1266--1284",
}

@article{谷中 瞳2021,
  title={JSICK: 日本語構成的推論・類似度データセットの構築},
  author={谷中 瞳 and 峯島 宏次},
  journal={人工知能学会全国大会論文集},
  volume={JSAI2021},
  number={ },
  pages={4J3GS6f02-4J3GS6f02},
  year={2021},
  doi={10.11517/pjsai.JSAI2021.0_4J3GS6f02}
}

Contributions

Thanks to Hitomi Yanaka and Koji Mineshima for creating this dataset.