Bengali NER for Person

In this repository, Bengali NER Model is trained and evaluated using two datasets. Used datasets are -

• Dataset-1
• Dataset-2

This raw datasets can be found in data folder.

Data Preprocessing

These datasets are not directly used for model training. As the datasets contain multiple labels, at first the labels are transformed in binary form. Also, it was taken care of that the space separated parts of a sentence align with the labels. If a word is a name or part of a name that is labeled 1 otherwise 0.

Notebook for data preprocessing can be found in notebooks folder.

processed_data folder contains balanced and unbalanced folders. Both this folder contains train.json, valid.json and test.json files. In balanced folder train.json file contains some repeated examples of those sentences which contain at least one name.

Here is the data distribution:

Here is an example of finally used data for training-

sentence = "আব্দুর রহিম নামের কাস্টমারকে একশ টাকা বাকি দিলাম"
label =       1   1   0      0      0   0   0    0

Train & Validation

Run train.py to train the model and run test.py to evaluate the model. BanglaBert is used here. Make sure to load the latest weight.

Also, you can run this Notebook to train and evaluate the model. The model is trained for three epochs, and the metric used to evaluate the model is the F1 score. The best F1 score was 0.8992.

Prediction

predictions folder contains prediction.csv file. Here is an example of model's prediction:

sentence = "এরকম একজন হারিয়ে যাওয়া মনীষী কিশোরগঞ্জের আনন্দমোহন বসু ।"
prediction = "আনন্দমোহন বসু"

Follow Paths

Bengali_NER_for_Person/
|-- data/
    |-- raw_data/
    |-- processed_data/
|-- notebooks/
    |-- bengali-ner.ipynb
    |-- processing_data.ipynb
|-- predictions/
    |-- predictions.csv
|-- utils/
    |-- helper.py
    |-- preprocessor.py
    |-- trainer.py
|-- requirements.txt
|-- train.py
|-- test.py