IndoLAMA

IndoLAMA is a knowledge probing benchmark for Indonesian language models. Processing of IndoLAMA dataset follows Petroni et al. 2019

Versions

This repository contais 3 variants of IndoLAMA. All variants are built from IndoWiki:

• IndoLAMA: Probing dataset taken from the transductive training split of IndoWiki (indowiki_transductive_train.txt). This means that any model trained with IndoWiki would have encountered these dataset. Useful to determine whether the trained model can recall factual knowledge properly.
• IndoLAMA-Test-Only: Probing dataset taken from the transductive test split of IndoWiki (indowiki_transductive_test.txt). Knowledge in this dataset is not present in the training split. However, entities (subject/object) that appear on this dataset may also appear in the training split.
• IndoLAMA-Inductive: Probing dataset taken from the inductive test split of IndoWiki (indowiki_inductive_test.txt). No entities present on this knowledge probing dataset should appear on the inductive training split of IndoWiki. This dataset is meant to test if a model is able to "induce" knowledge on unseen entities.

Knowledge Source

This benchmark uses IndoWiki as it's knowledge source. IndoWiki is a knowledge-graph dataset based on Wikipedia Bahasa Indonesia and WikiData. The method of acquiring factual knowledge from IndoWiki follows LAMA's preprocessing step for T-Rex dataset:

""" We consider 41 Wikidata relations and subsam- ple at most 1000 facts per relation. As with the Google-RE corpus, we manually define a tem- plate for each relation """

This repository follows the list of Wikidata relations used by LAMA.

Templates

The templates for each relation is manually defined. Below is the template definition for each predicate: