Estimating Uncertainty in Natural Language Processing

Git repository for the master's thesis "Estimating Uncertainty in Natural Language Processing" by Nils Grünefeld at MaiNLP group, CIS, LMU Munich

Repository Structure:

In principle, the python scripts in /scripts/ each cover one experiment and generate its results, which are saved in /data/.

/shell-scripts/ contains shell scripts each corresponding to an experiment python script, which are used to execute the experiment using SLURM.

The experimental results are then analyzed using the Jupyter notebooks in /notebooks/, with the generated figures being saved in /figures/.

The toy experiment experiments are the exception, where the entire experiment is performed in the corresponding notebook (and the resultung models saved in /notebooks/saved_experiments/).

/utils/ contains much of the substantial code used in the experiments.

• /utils/utils.py contains the gradient uncertainty method itself as well as the code to load all used datasets from huggingface and some general utility functions.
• /utils/gpt.py contains every usage of GPT, i.e. to rephrase text and evaluate model answers.

/slurm-outputs/ contains log files generated by SLURM for every run experiment.

requirements.txt contains the required packages to reproduce the experimental data (generally speaking all the python scripts), and requirements_eval.txt contains the required packages for the data analysis (generally speaking all the notebooks, including the toy experiment notebooks).