USFQ-MSDS-Accusatory-Comments-Classification

This repository is intended to store the necessary files for the execution, replication, and analysis of the "Recognition of Accusatory Comments in Ecuadorian Public Procurement Processes Using a Hybrid Approach: Unsupervised and Supervised Learning" project.

First, we will explain the structure and characteristics of the files, as this will help in understanding the tasks to be performed with the repository’s contents.

File Extensions:

This repostitory contains the next file extensions:

• .py - The .py files form the foundation of the project’s execution. All execution flows and logic were developed in Python and can be run either from .py scripts or .ipynb notebooks.
• .ipynb - The .ipynb files are analysis files. Only contains results.
• .sh - Some pipelines could be organized and executed using bash scripting. The bash scripts contains python executions.
• .txt - Files on txt are only used for parametrization (used on a few of python scripts)
• .csv - Files on csv contains data - Could be data to train the project or log data. The log data saved shows the results of the executions.

Nature of the files:

For this point, we only center on .py files. We have three types of files:

Tool Files:

This files are developed for a support reason. This project only have 2 tool files:

• tool_create_base_paths.py - This tool file creates all the paths needed to execute the .py files for model training purposes. The execution of this file it's mandatory before execute other .py files; otherwise, we can see some errors and bugs related to some wrong paths on the filesystem.
• tool_val_gpu_usage.py - This tool file return the GPU usage from a host. The information given by this file is used to decide the GPU that will be executing cuda scripts.
• tool_gen_logs_perf.py - This tool allow user to save information about GPU/CPU usage on the training steps. This tool work with every script executed on python.

Util Files:

It coud say that this files are user defined libraries. In this files you can find all the functions that I developed to work in this project. There are a few util files:

• utils.py - general purpose functions for all the project. Aditionally, this file contains functions related with clustering using sklearn.
• utils_clas.py - this file contains functions about classification and some utilities to facilitate it.
• utils_cls_description.py - this file contains functions about cluster description and some related utilities.
• utils_llms.py - this file contains NLP/LLM functions; we can find fine tune processes, embeddings discover processes and other related functions.
• utils_log_data.py - this file contains LOG functions; we can find functions to extract GPU or CPU usage and options to save the execution time of training steps.
• utils_stats.py - this file contains general purpose functions to analyze data from predictions.

Train Files:

This files are python script to do some concrete action. This scripts use the utils files and other general purpose libraries. There are a few train files in this project:

• train_clas_model.py - The main functionality it's to train a random forest classifier and save this model on a route for future use.
• train_models.py - This file it's used to train cluster algorithms.
• train_word_embeddings.py - This file it's used to do some LLM/NLP processes, like fine tune or embeddings aquistion from text.

Execution Example:

To make this example useful, we follow a full example of training (a complete pipeline). All the commands on this point are bash scripts:

1. First, we create all the paths needed for future executions. (It's possible to execute the command without parameters, the default value will be path_routes.txt)

python3 tool_create_base_paths.py --file_path "path_routes.txt"

2. At this point we execute the tool file to watch the GPU usage, we send our training to the GPU with less usage patterns.

python3 tool_val_gpu_usage.py

3. Next, we make a fine tune of a w2v algorithm to use in next steps:

python3 train_word_embeddings.py --cud_device "1" --process "TR" --llm_algo "W2" --tipo "UN" --default_llm_algo "S" --obs "Fine Tune example"

This command generate a W2V fine tune model, with the unsupervised dataset of the project, taking as a reference the "S" default model and putting on logs the observation value "Fine tune example".

4. Fill the ./language_models/llms_models.csv with the name of the model and put this value as "S" on column default.

5. Generate embeddings from the pretrained model. Depending on the type of default the execution take one or another.

python3 train_word_embeddings.py --cud_device "1" --process "EM" --llm_algo "W2" --tipo "UN" --default_llm_algo "S" --obs "Embeddings example"

This command generate a CSV file with the original dataframe added new columns with 1000 text embedding columns. It's important that the fine tune dataset was written on the ./language_models/llms_models.csv. The name of the out file will be:

• df_supervised_X - for "SU" --tipo
• df_unsupervised_X - for "UN" --tipo

The "X" value will be an integer that increments constantly. If the last execution save "df_supervised_3", the actual execution will save "df_supervised_4". We can recognize a particular execution based on the log_files.

6. Execute a clusterization method taking a particular dataset.

python3 train_models.py --dea_alg_multi "1" --max_clust "15" --algorithm "gmm" --clst_param1 "diag" --clst_param2 "500" --llm_algo_train "W2V_BASE" --dataset "df_unsupervised_w2v_6.csv" --scaler "1"

This command executes a GMM clustering, with a diag covariance and max 15 clusters with the csv "df_unsupervised_w2v_6.csv". The embeddings will be recognized automatically.

7. Execute a clasification algorithm:

python3 train_clas_model.py --dea_alg_multi "1" --k_clust "5" --algo_clas "R_Forest" --algo_clst "GMM" --algo_vec_phrase "w2v" --dataset "df_supervised_w2v_1.csv" --dis_smote "0" --dis_clus_filter "0" --obs "Observation for training"

This command execute a Random Forest Classifier with the "df_supervised_w2v_1.csv" dataset applying a GMM model of 5 clusters. Then find the most accusatory cluster and make classification about this. Aditionally, the execution will use smote and make a cluster filter (over most accusatory).

8. Analyze models trained on ipynb files or other python scritps.