Similarity comparison

Introduction of some improtant terms:

• Cosine similarity is a mathematical calculation that find the similarity between two words or sentences but in this code the similarity has been found between two words only that has been prompted by the user.


• TF_IDF(term Frequency Inverse Frequency) is a mathematicle calculation that finds the importance of each word in the document.
Term Frequency (TF) : It represents the frequency of a term within a document relative to the total number of terms in that document.
Mathematically it can be represented as follows:

   TF = (Number of times term t appears in a document)/(Number of terms in the document) 


Inverse document frequency (IDF): It is a measure used in information retrieval and text mining to quantify the importance of a term in a collection of documents.
Mathematically it can be represented as follows:


   Inverse document frequency(IDF) = log(N/n)
 
where N is the number of documents and n is the number of documents a term t has appeared in.


So as a conclusion:
Term Frequency-Inverse Document Frequency (TF-IDF) is a numerical representation that combines the local importance of a term within a document (TF) with its global significance in the collection (IDF).


    TF-IDF = TF*IDF



• Word Embedding is a technique used in natural language processing (NLP) to represent individual words as real-valued vectors in a predefined vector space. So, it's a way that make word representation in a way that can have a meaning of each word but a low-dimentional one.


• BERT(Bidirectional Encoder Representations from Transformers) is a pre-trained language model (transfer learning concept) that is designed to use pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. So, it uses the representation of the words to undestand the fully meaning of each word in a better performance that the other used techniques becuase it take the semantic meaning of each word by taking advantage of the place of the word and it's contextual location.

  Please note that the prompted words are words that exist inside the dataset.

About the used dataset:

• Dataset is collection of Arabic articles that can be used in different Arabic NLP tasks . The articles were collected using Python scripts written specifically from mawdoo3 website.

Mainly the dataset has two columns : [Title, article] that contains a total number of 5000 articles.

Requirements:

• First of all you need to extract the compressed file "Dataset!"

import zipfile as zf
files = zf.ZipFile("Arabic_dataset.zip", 'r')
files.extractall("Arabic_dataset")
files.close()

• Now you need to install the required libraries:
     torch    transformers
     pandas
     numpy
     scikit-learn
     regex
     strings
     python-math
     python3.x

  check the requirment.txt file.

• note: if your device is slow you can use to run the code on it. Also, activate the GPU via clicking on the runtime from the toolbar >>> change runtime type >>> Hardware accelerator >>>> GPU.

Now you're ready to run your code. It's provided with more than one type such as .ipynb extention which is compatible with jupyter notebook & .py extention which is compatible with python file.