Semantic Proto-role Labelling using Natural Language Inference

This Repository contains a viable, novel method for Semantic Proto-role Labelling (SPRL) using Natural Language Inference (NLI). We provide formatted SPRL datasets, a SPRL-finetuned variant of roberta-large-mnli and a logistic regressor to solve the binary classification problem of SPRL. By predicting the entailment of a given property set meant to identify agent- and patientlike qualities of a given argument in a given input sentence, we create an entailment vector functioning as the SPRL model's input. We evaluate the resulting models using state-of-the-art frameworks, baselines and various robustness tests and observe performance in line with the state-of-the-art.

Table of Contents

• Getting Started
• Achievements
• Concept
  • SPRL-Model
• Data
• Training
  • RoBERTa
  • SPRL-model
• Evaluation
  • Quantitative
  • Robustness
  • Interpretability
• Conclusion
  • Future Work
• Acknowledgements
  • License

Authors

• Marlon Dittes, marlon.dittes@stud.uni-heidelberg.de
• Shai Peretz, dorian.peretz@stud.uni-heidelberg.de
• Iva Andreeva, iva.andreeva@stud.uni-heidelberg.de

Getting Started

In order to get started, clone the repository as follows:

• git clone https://github.com/Shaip161/SPRL.git --recursive

Use --recursive to clone the roberta-large-mnli submodule as well.

After cloning the repository, create a Python virtual environment in the project directory. This isolates the project dependencies from the global Python environment. Use the following command:

• python -m venv my_venv

After that, activate the virtual envirnment using the commands:

• On Windows: my_venv\Scripts\activate
• On macOS and Linux: source my_venv/bin/activate

With the virtual environment activated, install all required Python packages specified in the requirements.txt file:

• pip install -r requirements.txt

Concept

In previous work, sets of 14 to 18 properties were applied to target arguments in SPRL tasks. On this basis, sentences were generated with statements postulating the observed argument have that property. We use these sentences as entailment hypotheses and use an NLI model to classify the entailment of this hypothesis in the observed sentence. The NLI model is finetuned on Likert-Scale data: Human annotators annotate each hypothesis with a value of 1 - 5. These annotations infer gold labels for the NLI task.

SPRL-Model

Given the likelihoods of entailment and contradiction for each hypothesis and a gold label based on VerbNet thematic roles, the SPRL model regresses on a weight vector to solve the binary classification into proto-agent and proto-patient.

Data

We used Reisinger's SPR1 and White's SPR2 dataset to create our own datasets (json files) for later use:

Generally, reis.json, reis_labelled.json, spr1.json, spr2.json, spr2_1.json, spr2_no_pilot.json all consist of the following information:

• applicable: Whether the property is applicable in principle to the argument in question; either True or False.
• property: The proto-role property being annotated.
• label: The 5-point Likert scale annotation of the property, where 1 means very unlikely and 5 means very likely.
• sentence: The base sentence, which we will later use as the premise.
• split: The training split used in training the model, either train, dev, or test.
• arg: The argument in the sentence.
• pred: The predicate in the sentence.
• gram_func: The grammatical function of the argument. subj, obj or other for Reisinger and nsubj, nsubjpass, dobj or iobj for White.

Reis.json

reis.json was created using Reisinger's SPR1 data to finetune RoBERTa. It additionally contains the following entries:

• roleset: The propbank roleset of the predicate.
• arg_num: The propbank argument label; either 0, 1, 2, 3, 4 or 5.

It consists of 6363 entries using 18 different properties.

Reis_labelled.json

reis_labelled.json is the subset of entries in reis.json on which we could find a VerbNet entry for the giving roleset using NLTK. We use this data to train weights for the different property features and also to evaluate how good a model is at the SPRL task. It additionally contains the following entry:

• is_agent: Denotes if the given argument is of proto-agent status in the given sentence; either 0 or 1. It consists of 2715 entries using the same properties as reis.json.

spr1.json, spr2.json, spr2_1.json

spr1.json, spr2.json, spr2_1.json were created using White's SPR2 data. They are mainly used for finetuning RoBERTa and additionally have the following entry:

• ispilot: Whether the annotation was part of a pilot or not; either True or False. Pilot annotations pass the filters introduced by Reisinger et al. They respectively consist of 92, 2850 and 1396 entries. Between the three there are however duplicates. spr1.json uses 16 different properties, whereas the remaining two use 14 properties.

spr2_no_pilot.json

spr2_no_pilot.json consists of all spr2.json entries with ispilot = False. It is used for ablation studies. It consists of 2758 entries.

Duplicates

There are sentences which got annotated by multiple annotators. To not have duplicate (premise, hypothesis) pairs and decrease overall finetuning duration, we also use variations of the above datasets where we average over all the labels for a given property.

Weight training

To train the weights of our different properties we created files of the format "model_predictions.json" containing 2715 datapoints which each consist of the following entries:

• is_agent: Denotes if the given argument is of proto-agent status in the given sentence; either 0 or 1.
• split: The training split used in training the model, either train, dev, or test.
• roberta_entailments: The likelihood of entailment which the model returned for a (premise, hypothesis) pair on a given property.
• roberta_contradictions: The likelihood of contradiction which the model returned for a (premise, hypothesis) pair on a given property.

Training

We finetuned RoBERTa and trained our SPRL model separately, as the SPRL training data had to include the entailment predictions. Due to the sequential nature of this training process, it was not only computationally intensive, but also time-intensive. Thus, a bug which impacted performance of the SPRL model greatly was noticed shortly before the project deadline.

RoBERTa

As our base model we used roberta-large-mnli, which is the RoBERTa large model fine-tuned on the Multi-Genre Natural Language Inference (MNLI) corpus. We use this model to maximize performance on the NLI task needed during the finetuning process.

Procedure

To generate train, test and dev data with gold labels from our own datasets (json) which use likert scale data, we did the following procedure:

We iterate through all datapoints in our dataset. We create the pair of (premise, hypothesis) as follows:

• premise: The sentence of the datapoint.
• hypothesis: Created using template hypotheses which use the argument (ARG) and predicate (PRED) of the datapoint. Example for "instigation": "ARG caused PRED to happen."

If a property is deemed applicable in its context, we denote the gold label as 'entailment' if the likert label is >= 4 for the given property and 'contradiction' if the likert label is < 4. If a property is not deemed applicable, we denote the gold label as 'neutral'.

We chose to handle gold labels this way, since non-applicable properties don't provide a feasible likert label for us to use.

Training Arguments

We finetuned the roberta-large-mnli model using Hugging Face's transformer library Trainer class using the following TrainingArguments:

• num_train_epochs=3 Note: We used 2 epochs on roberta_retrained_non-averaged due to the finetuning process already taking a lot of time.
• per_device_train_batch_size=16
• per_device_eval_batch_size=64
• warmup_steps=500
• weight_decay=0.01

Models

We finetuned 5 different models:

• roberta_averaged: Uses averaged versions of reis.json, spr1.json, spr2.json, spr2_1.json.
• roberta_non-averaged: Uses non-averaged reis.json, spr1.json, spr2.json, spr2_1.json.
• roberta_reisinger: Uses averaged version of reis.json.
• roberta_white: Uses averaged version of spr1.json, spr2.json, spr2_1.json.
• roberta_no-pilot: Uses averaged version of spr2_no_pilot.json.

SPRL-model

Given a model and an instance of our reis_labelled.json, we tackle the SPRL task in the following way:

We use the given model to compute entailment and prediction likelihoods for all 18 properties. We then calculate the dot product of these 36 features with 36 weights, whilst adding an bias. After that, we use the sigmoid function on the result; a value >= 0.5 returns proto-agent as predicted, a value < 0.5 returns proto-patient as predicted.

To get the weights and bias, we created "model_predictions.json" files as mentioned in Data. We then used the scikit-learn library to create a Logistic Regression linear model for training the weights and bias on our train data. We also tried a Support Vector Machine model to make sure we get a good linear decision boundary whilst using Logistic Regression, which turned out to produce the same accuracy. We additionally directly computed weights using the likert scale data from our dataset, but this turned out to overall have significantly worse accuracy.

Evaluation

In the following section we will introduce 3 evaluation techniques and results in order to objectively compare the trained models with one another.

Quantitative

The Quantitative technique includes:

• Overall accuracy, to calculate the proportion of correct predictions across all categories.
• The F1 score, which balances precision and recall, reflecting the model's reliability in positive case identification.
• Average Log Loss in order to measure prediction certainty, with lower values indicating higher accuracy.
• Class-specific accuracy to assess the model's performance in individual categories, identifying potential biases or weaknesses.

The results from our different models:

Robustness

To test the robustness of our models, we have carried out:

• An ablation study, where we systematically remove a specific part of our models to understand its impact on performance. We did this by removing all dataset entries, where the isPilot tag set to true from the SPR2 dataset. This is because these sentences were particularly well suited for entailment questions. By removing them, we have not only reduced the dataset on which the model can train, but we have also removed the sentence where the entailment questions are ideal.
• Adversarial testing. Here we want to evaluate how our models perform against carefully crafted adversarial examples to test the limits of their understanding. These hand-crafted sentences include, for example, various biases such as gender bias, where a male is generally more likely to be a proto-agent than a female.
• Out of distribution tests, where we test the models on data that is significantly different from the training distribution. This data includes hand-crafted nonsensical sentences that were not used in the training of our models.

The results of the Adversial Evaluation method using the wights obtained from logistical regression:

The control dataset result

The bias dataset results:

The results of the Out of Distribution method using the wights obtained from logistical regression:

Interpretability

To understand the decision-making of our SPRL-model, we inspect the weights learned from training: As the entailment of certain properties indicates proto-agenthood, those properties recieve large, positive entailment weights. In turn, proto-patient-related property entailment weights recieve large, positive weights as well. By this system, a largely coherent mapping to proto-agent and proto-patient properties can be extracted.

Additionally, we observe token importance in our RoBERTa variant models to understand which part of an input sentence is used to infer the queried properties. We perform these observations using transformers-interpret on multiple examples for multiple properties, including sentience and change of state.

Using ferret, we are able to not only explain the entailment prediction of the property sentience for "He listens impassively.", but also evaluate the explanation by observing correlation between state-of-the-art explainability methods.

Conclusion

We conclude that the approach to SPRL using NLI has merit, as one of our non-averaged model variants can compete with state-of-the-art baselines on similar datasets. However, our method, especially using roberta-large-mnli is immensely computationally intensive and requires large amounts of data. We therefore see great potential for improvement.

Future Work

• A refinement of the entailment hypotheses to conform to grammar rules may constitute a large improvement in entailment accuracy, which can incite improvement in the SPRL model. -- Note: Shortly before the project deadline, a bug was found in the averaging algorithm, which impacted performance greatly: applicability values were assumed True/False instead of the actual yes/no values.
• As by far not all sentences in the Reisinger dataset had corresponding PropBank-roleset - VerbNet-class pairs in nltk, a large amount of data could not be labelled. Using newer or different resources may increase the available amount of data.
• The PropBank predominantly contains financial data, and may therefore be suboptimally representative of the English language in general. The use of a different dataset, which matches the target distribution better, may improve performance and may be more robust towards out-of-distribution data.

Acknowledgements

We thank our Professor, Dr. Anette Frank, as well as the Institute for Computational Linguistics of the Heidelberg University, for the support we were given and especially the access to resources like the PennTreeBank. This work would not have been possible if not for the written guidance of Reisinger et al. and White et al., who provided the SPR1 and SPR2 datasets. This project additonally powered by huggingface and its provided libraries, the PennTreeBank, PropBank, VerbNet and nltk.

License

We give out our work under the MIT License.