Repository Content

This repository contains the files and scripts created during my thesis, aimed at building a dataset of italian laws and quizzes related to the italian legislative field. It provides tools for data extraction, processing, and analysis, alongside experiments conducted using LLMs.

Repository Description

The repository's main root is composed of two main folders "Data" and "Documents", and a bunch of informational files, like the "requirements.txt", which contains all the libraries of my executions environement. The "Data" folder contains all the scripts used during the process, some for web scraping, others for data manipulation and file creation, others for LLM fine-tuning and evaluation. The "Documents" folder is divided in two main sub-folders: "Downloaded" and "Generated", the first one contains files i downloaded online and then elaborated, while the "Generated" folder contains all the documents and dataset i generated, there are also some folders regarding a training model for law retrieval from a Milvus vector database.

Executions

These are the main exectution scripts and how they should be called.

Data Extraction

For data extraction there are two main script: "Laws extraction.ipynb" which require a web-driver like Chromedriver or Firefox driver to execute a scraper with Selenium on the Normattiva website, the second script is "Quiz extractor", for quizzes extraction from different sources. As ipynb will be just needed to execute the cells.

Data Manipulation

The main data manipulation script is "Referenecs extraction from quiz.ipynb", this file's cells contains extract one kind of source (italian constitution, penal code,...), all the reference from the quizzes dataset using regex, will then link the obtained references with the actual laws from the laws dataset to create the "references_merged_final.csv" file. There is also the "Upload datasets.ipynb" file containing scripts to split in training, test and validation set, and upload the csv generated files to HuggingFace.

LLM Training And Evaluation

In the thesis there have been several evaluations done, the main one are listed here:

Native Response Analysis

These scripts goal is to calculate the accuracy of the model natively, for small models there is "FineTuning small models.sh" while for big ones "FineTuning big models.sh", inside of the sh script the parameter "dataset_name" should be "IGieckI/fine_tuning_dataset" and "--no-article" parameter in the end should not be commented, moreover an huggingface model should be passed as follow:

./FineTuning\ big\ models.sh "meta-llama/Llama-3.1-8B-Instruct"

./FineTuning small models.sh "facebook/mbart-large-50"

Response with Legislative Text

This experiment is called as the previous one, but the "--no-article" parameter should be commented.

Response in English

This experiment is called as the previous one, but the "dataset_name" parameter should be "IGieckI/fine_tuning_dataset_en".

Response with Context

This experiment needs a context to be generated with the file "Generate context.py" giving the generating model as parameter:

./Generate\ context.py "Equall/Saul-7B-Instruct-v1"

this will generate a csv file, which should be uploaded with the "Upload datasets.ipynb" file. I have already uploaded contextes for Llama 3.1 and Saul, which are the default dataset in the following scripts. Then, to evaluate the models with the script call "FineTuning context small models.sh" or "FineTuning context big models.sh", using an HuggingFace geneartive answer model as parameter, for example:

./FineTuning\ context\ big\ models.sh "meta-llama/Llama-3.1-8B-Instruct"

It is possible to change the input dataset with the "dataset_subset" and "dataset_name" parameters.

RAG Precision

A RAG had also been tried to retrieve the referred law from the context of the question.

Full Context Retrieval

This experiment uses the entire context of the question for the retrieval of the law, and is performed in two phases:

1. Generation of the vector database with:

./Tokenization\ and\ Milvus\ insertion.py

this script is using "BAAI/bge-m3" as default tokenizer model.

2. Call the evaluation script:

./Calculate\ RAG\ precision.py

as the previous script, this is using "BAAI/bge-m3" as default tokenizer model.

Full Context Retrieval with Training

This experiment is executed as the previous one, but the tokenizer model used is trained before (using BAAI/bge-m3 as default one), the steps are:

1. Be sure to have executed all cells of "Referenecs extraction from quiz.ipynb" before, to generate all training files.

2. Train the model with:

./Train\ model\ on\ laws\ search.py

3. Generate the vector database with:

./Tokenization\ and\ Milvus\ insertion.py "path to the trained model"

the parameter need to be the retrieval trained model.

4. Call the evaluation script:

./Calculate\ RAG\ precision.py "path to the trained model"

as the previous script, the parameter need to be the retrieval trained model.

Retrieval Using Legislative References

The experiments with the formatted input as very similar, and executed as follow.

For the "source | law_number | year | text" one:

1. Generation of the vector database with:

./Tokenization\ and\ Milvus\ insertion\ format\ v2.py

this script is using "BAAI/bge-m3" as default tokenizer model.

2. Call the evaluation script:

./Calculate\ RAG\ precision\ v2.py

as the previous script, this is using "BAAI/bge-m3" as default tokenizer model.

For the "source | law_number | year" one:

1. Generation of the vector database with:

./Tokenization\ and\ Milvus\ insertion\ format\ v3.py

this script is using "BAAI/bge-m3" as default tokenizer model.

2. Call the evaluation script:

./Calculate\ RAG\ precision\ v3.py

as the previous script, this is using "BAAI/bge-m3" as default tokenizer model.