CALAME-PT

Context-Aware LAnguage Modeling Evaluation for Portuguese

CALAME-PT is a PT benchmark composed of small texts (contexts) and their respective last words. These contexts should, in theory, contain enough information so that a human or a model is capable of guessing its last word - without being too specific and/or too ambiguous.

Composition

CALAME-PT is composed of 2 "sets" of data - handwritten and generated.

• Handwritten Set: contains 406 handwritten samples by Portuguese native speakers;
• Generated Set: contains 1670 samples generated by GPT-3.5, posteriorly submitted to a human-review.

Automatic Generation

To generate the samples for the generated set, documents were randomly sampled from 3 post-processed data sources: Portuguese Wikipedia (PTWIKI), OSCAR and Arquivo.pt. These datasets correspond to the ones mentioned and used by GlorIA. Posteriorly, GPT-3.5 was prompted with each document, in order to rewrite/summarize then, and to perform anonimization steps on entities (people, companies, etc).

Then the generated samples were submitted to a human review where 1) bad samples were removed 2) some rewrites were performed to ensure quality and avoid removing an entire sample 3) perform anonimization steps by hand.

How to use

Make sure your Datasets package is updated!

from datasets import load_dataset
>>> dataset = load_dataset("NOVA-vision-language/calame-pt", "all")
>>> dataset_handwritten_only = load_dataset("NOVA-vision-language/calame-pt", "handwritten")
>>> dataset_generated_only = load_dataset("NOVA-vision-language/calame-pt", "generated")