Dataset Card for MultiLegalPile: A Large-Scale Multilingual Corpus for the Legal Domain

Table of Contents

• Table of Contents
• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage:
• Repository:
• Paper: MultiLegalPile
• Leaderboard:
• Point of Contact: Joel Niklaus

Dataset Summary

The Multi_Legal_Pile is a large-scale multilingual legal dataset suited for pretraining language models. It spans over 24 languages and five legal text types.

Supported Tasks and Leaderboards

The dataset supports the tasks of fill-mask.

Languages

The following languages are supported: bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

Dataset Structure

It is structured in the following format: type -> language -> jurisdiction.jsonl.xz

type is one of the following:

• caselaw
• contracts
• legislation
• other
• legal_mc4

legal_mc4 is a subset of the other type but is listed separately so it can be easily excluded since it is less permissively licensed than the other types.

Use the dataset like this:

from datasets import load_dataset

config = 'en_contracts'  # {language}_{type}
dataset = load_dataset('joelniklaus/Multi_Legal_Pile', config, split='train', streaming=True)

'config' is a combination of language and text_type, e.g. 'en_contracts' or 'de_caselaw'. To load all the languages or all the text_types, use 'all' instead of the language or text_type (e.g., ' all_legislation').

Data Instances

The file format is jsonl.xz and there is one split available ("train").

The complete dataset (689GB) consists of four large subsets:

• Native Multi Legal Pile (112GB)
• Eurlex Resources (179GB)
• Legal MC4 (106GB)
• Pile of Law (292GB)

Native Multilingual Legal Pile data

Eurlex Resources

See Eurlex Resources for more information.

Legal-MC4

See Legal-MC4 for more information.

Pile-of-Law

See Pile-of-Law for more information.

Data Fields

[More Information Needed]

Data Splits

[More Information Needed]

Dataset Creation

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

[More Information Needed]

Who are the source language producers?

[More Information Needed]

Annotations

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

[More Information Needed]

Citation Information

@misc{niklaus2023multilegalpile,
      title={MultiLegalPile: A 689GB Multilingual Legal Corpus}, 
      author={Joel Niklaus and Veton Matoshi and Matthias Stürmer and Ilias Chalkidis and Daniel E. Ho},
      year={2023},
      eprint={2306.02069},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contributions

Thanks to @JoelNiklaus for adding this dataset.