Dataset Card for "super_glue"

Table of Contents

• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage: https://super.gluebenchmark.com/
• Repository: More Information Needed
• Paper: https://arxiv.org/abs/1905.00537
• Point of Contact: More Information Needed
• Size of downloaded dataset files: 58.36 MB
• Size of the generated dataset: 249.57 MB
• Total amount of disk used: 307.94 MB

Dataset Summary

SuperGLUE (https://super.gluebenchmark.com/) is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard.

Supported Tasks and Leaderboards

More Information Needed

Languages

More Information Needed

Dataset Structure

Data Instances

axb

• Size of downloaded dataset files: 0.03 MB
• Size of the generated dataset: 0.24 MB
• Total amount of disk used: 0.27 MB

An example of 'test' looks as follows.

axg

• Size of downloaded dataset files: 0.01 MB
• Size of the generated dataset: 0.05 MB
• Total amount of disk used: 0.06 MB

An example of 'test' looks as follows.

boolq

• Size of downloaded dataset files: 4.12 MB
• Size of the generated dataset: 10.40 MB
• Total amount of disk used: 14.52 MB

An example of 'train' looks as follows.

cb

• Size of downloaded dataset files: 0.07 MB
• Size of the generated dataset: 0.20 MB
• Total amount of disk used: 0.28 MB

An example of 'train' looks as follows.

copa

• Size of downloaded dataset files: 0.04 MB
• Size of the generated dataset: 0.13 MB
• Total amount of disk used: 0.17 MB

An example of 'train' looks as follows.

Data Fields

The data fields are the same among all splits.

axb

• sentence1: a string feature.
• sentence2: a string feature.
• idx: a int32 feature.
• label: a classification label, with possible values including entailment (0), not_entailment (1).

axg

• premise: a string feature.
• hypothesis: a string feature.
• idx: a int32 feature.
• label: a classification label, with possible values including entailment (0), not_entailment (1).

boolq

• question: a string feature.
• passage: a string feature.
• idx: a int32 feature.
• label: a classification label, with possible values including False (0), True (1).

cb

• premise: a string feature.
• hypothesis: a string feature.
• idx: a int32 feature.
• label: a classification label, with possible values including entailment (0), contradiction (1), neutral (2).

copa

• premise: a string feature.
• choice1: a string feature.
• choice2: a string feature.
• question: a string feature.
• idx: a int32 feature.
• label: a classification label, with possible values including choice1 (0), choice2 (1).

Data Splits

axb

axg

boolq

cb

copa

Dataset Creation

Curation Rationale

More Information Needed

Source Data

Initial Data Collection and Normalization

More Information Needed

Who are the source language producers?

More Information Needed

Annotations

Annotation process

More Information Needed

Who are the annotators?

More Information Needed

Personal and Sensitive Information

More Information Needed

Considerations for Using the Data

Social Impact of Dataset

More Information Needed

Discussion of Biases

More Information Needed

Other Known Limitations

More Information Needed

Additional Information

Dataset Curators

More Information Needed

Licensing Information

The primary SuperGLUE tasks are built on and derived from existing datasets. We refer users to the original licenses accompanying each dataset, but it is our understanding that these licenses allow for their use and redistribution in a research context.

Citation Information

If you use SuperGLUE, please cite all the datasets you use in any papers that come out of your work. In addition, we encourage you to use the following BibTeX citation for SuperGLUE itself:

@article{wang2019superglue,
  title={Super{GLUE}: A Stickier Benchmark for General-Purpose Language Understanding Systems},
  author={Alex Wang and Yada Pruksachatkun and Nikita Nangia and Amanpreet Singh and Julian Michael and Felix Hill and Omer Levy and Samuel R. Bowman},
  journal={arXiv preprint 1905.00537},
  year={2019}
}
@inproceedings{clark2019boolq,
  title={{B}ool{Q}: Exploring the Surprising Difficulty of Natural Yes/No Questions},
  author={Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina},
  booktitle={Proceedings of NAACL-HLT 2019},
  year={2019}
}
@inproceedings{demarneffe:cb,
  title={{The CommitmentBank}: Investigating projection in naturally occurring discourse},
  author={De Marneffe, Marie-Catherine and Simons, Mandy and Tonhauser, Judith},
  note={To appear in proceedings of Sinn und Bedeutung 23. Data can be found at https://github.com/mcdm/CommitmentBank/},
  year={2019}
}
@inproceedings{roemmele2011choice,
  title={Choice of plausible alternatives: An evaluation of commonsense causal reasoning},
  author={Roemmele, Melissa and Bejan, Cosmin Adrian and Gordon, Andrew S.},
  booktitle={2011 AAAI Spring Symposium Series},
  year={2011}
}
@inproceedings{khashabi2018looking,
  title={Looking beyond the surface: A challenge set for reading comprehension over multiple sentences},
  author={Khashabi, Daniel and Chaturvedi, Snigdha and Roth, Michael and Upadhyay, Shyam and Roth, Dan},
  booktitle={Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  pages={252--262},
  year={2018}
}
@article{zhang2018record,
  title={{ReCoRD}: Bridging the Gap between Human and Machine Commonsense Reading Comprehension},
  author={Sheng Zhang and Xiaodong Liu and Jingjing Liu and Jianfeng Gao and Kevin Duh and Benjamin Van Durme},
  journal={arXiv preprint 1810.12885},
  year={2018}
}
@incollection{dagan2006pascal,
  title={The {PASCAL} recognising textual entailment challenge},
  author={Dagan, Ido and Glickman, Oren and Magnini, Bernardo},
  booktitle={Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment},
  pages={177--190},
  year={2006},
  publisher={Springer}
}
@article{bar2006second,
  title={The second {PASCAL} recognising textual entailment challenge},
  author={Bar Haim, Roy and Dagan, Ido and Dolan, Bill and Ferro, Lisa and Giampiccolo, Danilo and Magnini, Bernardo and Szpektor, Idan},
  year={2006}
}
@inproceedings{giampiccolo2007third,
  title={The third {PASCAL} recognizing textual entailment challenge},
  author={Giampiccolo, Danilo and Magnini, Bernardo and Dagan, Ido and Dolan, Bill},
  booktitle={Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing},
  pages={1--9},
  year={2007},
  organization={Association for Computational Linguistics},
}
@article{bentivogli2009fifth,
  title={The Fifth {PASCAL} Recognizing Textual Entailment Challenge},
  author={Bentivogli, Luisa and Dagan, Ido and Dang, Hoa Trang and Giampiccolo, Danilo and Magnini, Bernardo},
  booktitle={TAC},
  year={2009}
}
@inproceedings{pilehvar2018wic,
  title={{WiC}: The Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations},
  author={Pilehvar, Mohammad Taher and Camacho-Collados, Jose},
  booktitle={Proceedings of NAACL-HLT},
  year={2019}
}
@inproceedings{rudinger2018winogender,
  title={Gender Bias in Coreference Resolution},
  author={Rudinger, Rachel  and  Naradowsky, Jason  and  Leonard, Brian  and  {Van Durme}, Benjamin},
  booktitle={Proceedings of NAACL-HLT},
  year={2018}
}
@inproceedings{poliak2018dnc,
  title={Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation},
  author={Poliak, Adam and Haldar, Aparajita and Rudinger, Rachel and Hu, J. Edward and Pavlick, Ellie and White, Aaron Steven and {Van Durme}, Benjamin},
  booktitle={Proceedings of EMNLP},
  year={2018}
}
@inproceedings{levesque2011winograd,
  title={The {W}inograd schema challenge},
  author={Levesque, Hector J and Davis, Ernest and Morgenstern, Leora},
  booktitle={{AAAI} Spring Symposium: Logical Formalizations of Commonsense Reasoning},
  volume={46},
  pages={47},
  year={2011}
}

Contributions

Thanks to @thomwolf, @lewtun, @patrickvonplaten for adding this dataset.