MarioKuzmanov/Thesis

Multilingual & Cross-Lingual NER Transfer for Slavic Languages

Jupyter Notebook

7 commits

updated Oct 5, 2025

See the code

README

BA Thesis

Multilingual and Cross-Lingual NER Transfer for Slavic Languages

1. Project Setup

1.1 Setup Virtual Environment

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-*.sh
conda create -n .venv python=3.12.4

1.2 Install Requirements

pip install -r requirements.txt

1.3 Download the data

   cd data
   git clone https://github.com/SlavicNLP/SlavicNER

Start with the experiments

2. Project Structure

BA Thesis

├── data # Cross-Lingual NEs corpus

├── related_work # Related Work

├── screenshots # Confusion Matrices for models and data distribution

├── src # implementations and experiments
      ├── data_analysis # Overview and Visualizations of the cross-topic data split
      ├── utils # Useful scripts (as notebooks) for preprocessing the data, preparing the supervised/unsupervised experiments and the confusion matrices
      ├── RNNs
      ├── GLiNER
      ├── XLM-RoBERTa-large
      ├── GLiNER
      ├── LLMs
      ├── scripts

3. Small peek at the results - Named Entity Recognition (NER)

N.B. Please refer to the thesis for a thorough results of all methods, or get the results yourself by running scripts/eval_ner.py with the correct file paths.

SPLIT_cross-topic - Fine-Tuning

Train on all languages from SPLIT_cross-topic_train

Test on all languages from SPLIT_cross-topic_test

per-model performance on SPLIT_cross-topic_test

Support = 13961	P-macro	R-macro	F1-macro	P-weighted	R-weighted	F1-weighted	P-micro	R-micro	F1-micro
XLM-RoBERTa-large (5 epochs)	87.83%	91.31%	89.51%	91.24%	93.51%	92.36%	91.12%	93.51%	92.30%
2-BiLSTM (10 epochs)	87.11%	80.86%	83.84%	89.97%	83.14%	86.42%	90.00%	83.14%	86.43%
2-BiGRU (10 epochs)	86.28%	79.28%	82.58%	88.49%	81.97%	85.10%	88.55%	81.97%	85.13%
2-BiLSTM (5 epochs)	81.68%	81.74%	81.48%	85.67%	83.57%	84.61%	85.16%	83.57%	84.36%
2-BiGRU (5 epochs)	83.85%	78.28%	80.75%	85.46%	81.73%	83.55%	85.45%	81.73%	83.55%
2-BiSRNN (10 epochs)	80.00%	74.08%	76.85%	83.56%	79.24%	81.35%	83.86%	79.24%	81.48%
2-BiSRNN (5 epochs)	75.42%	70.29%	72.61%	80.14%	76.98%	78.53%	80.54%	76.98%	78.72%

per-category performance of XLM-RoBERTa-large (5 epochs)

Category	Precision	Recall	F1	Support
EVT	82.17%	90.71%	86.23%	1163
LOC	95.58%	96.04%	95.81%	4575
ORG	87.29%	89.65%	88.45%	3516
PER	95.72%	97.73%	96.72%	3752
PRO	78.39%	82.41%	80.35%	955

per-language performance of XLM-RoBERTa-large (5 epochs)

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	94.08%	96.29%	95.16%	95.00%	95.98%	95.49%	2987
CS	92.22%	95.95%	94.01%	94.58%	96.94%	95.75%	1567
RU	84.36%	88.78%	86.49%	88.24%	91.71%	89.94%	3460
BG	88.49%	91.65%	90.02%	92.39%	94.74%	93.55%	2872
SL	82.56%	86.79%	84.55%	87.14%	90.84%	88.95%	1977
UK	83.34%	84.53%	83.88%	88.92%	89.16%	89.04%	1098

lang_id per-category performance of XLM-RoBERTa-large (5 epochs)
lang_id = PL

Category	Precision	Recall	F1	Support
EVT	92.74%	97.08%	94.86%	171
LOC	96.98%	97.06%	97.02%	1090
ORG	92.81%	93.04%	92.93%	805
PER	96.54%	96.97%	96.75%	661
PRO	91.34%	97.31%	94.23%	260

lang_id = CS

Category	Precision	Recall	F1	Support
EVT	84.25%	94.69%	89.17%	113
LOC	97.75%	97.42%	97.59%	581
ORG	91.48%	96.31%	93.83%	379
PER	97.50%	98.48%	97.99%	396
PRO	90.10%	92.86%	91.46%	98

lang_id = RU

Category	Precision	Recall	F1	Support
EVT	82.87%	90.17%	86.36%	295
LOC	93.42%	96.08%	94.73%	1123
ORG	83.35%	86.20%	84.75%	935
PER	95.15%	97.46%	96.29%	865
PRO	67.04%	73.97%	70.33%	242

lang_id = BG

Category	Precision	Recall	F1	Support
EVT	85.24%	93.09%	88.99%	304
LOC	97.08%	96.21%	96.64%	897
ORG	87.99%	91.18%	89.56%	635
PER	96.48%	99.10%	97.77%	886
PRO	75.64%	78.67%	77.12%	150

lang_id = SL

Category	Precision	Recall	F1	Support
EVT	70.64%	84.15%	76.81%	183
LOC	93.10%	96.13%	94.59%	491
ORG	81.17%	83.42%	82.28%	398
PER	93.55%	96.41%	94.96%	752
PRO	74.34%	73.86%	74.10%	153

lang_id = UK

Category	Precision	Recall	F1	Support
EVT	73.83%	81.44%	77.45%	97
LOC	94.68%	90.59%	92.59%	393
ORG	86.52%	88.19%	87.35%	364
PER	96.94%	98.96%	97.94%	192
PRO	64.71%	63.46%	64.08%	52

per-category performance of 2-BiLSTM (10 epochs)

Category	Precision	Recall	F1	Support
EVT	88.24%	85.81%	87.01%	1163
LOC	94.77%	90.34%	92.50%	4575
ORG	86.01%	78.84%	82.27%	3516
PER	92.38%	81.45%	86.57%	3752
PRO	74.14%	67.85%	70.86%	955

per-language performance of 2-BiLSTM (10 epochs)

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	91.90%	85.85%	88.69%	92.78%	85.14%	88.79%	2987
CS	89.73%	84.76%	87.11%	92.62%	86.47%	89.44%	1567
RU	82.25%	78.71%	80.39%	85.93%	81.04%	83.42%	3460
BG	89.35%	83.02%	86.02%	91.71%	85.86%	88.69%	2872
SL	81.45%	69.64%	74.61%	86.96%	74.91%	80.49%	1977
UK	91.77%	82.04%	86.27%	92.29%	87.25%	89.70%	1098

lang_id per-category performance of 2-BiLSTM (10 epochs)
lang_id = PL

Category	Precision	Recall	F1	Support
EVT	97.56%	93.57%	95.52%	171
LOC	95.56%	88.90%	92.11%	1090
ORG	91.23%	82.73%	86.78%	805
PER	93.81%	80.18%	86.46%	661
PRO	81.34%	83.85%	82.58%	260

lang_id = CS

Category	Precision	Recall	F1	Support
EVT	86.49%	84.96%	85.71%	113
LOC	96.55%	91.39%	93.90%	581
ORG	89.43%	87.07%	88.24%	379
PER	94.96%	80.81%	87.31%	396
PRO	81.25%	79.59%	80.41%	98

lang_id = RU

Category	Precision	Recall	F1	Support
EVT	86.36%	83.73%	85.03%	295
LOC	91.71%	88.69%	90.18%	1123
ORG	83.41%	74.76%	78.85%	935
PER	88.46%	81.50%	84.84%	865
PRO	61.33%	64.88%	63.05%	242

lang_id = BG

Category	Precision	Recall	F1	Support
EVT	88.89%	86.84%	87.85%	304
LOC	97.00%	93.76%	95.35%	897
ORG	86.15%	80.31%	83.13%	635
PER	92.67%	84.20%	88.23%	886
PRO	82.03%	70.00%	75.54%	150

lang_id = SL

Category	Precision	Recall	F1	Support
EVT	79.23%	79.23%	79.23%	183
LOC	93.35%	88.59%	90.91%	491
ORG	74.92%	62.31%	68.04%	398
PER	93.83%	78.86%	85.69%	752
PRO	65.93%	39.22%	49.18%	153

lang_id = UK

Category	Precision	Recall	F1	Support
EVT	95.56%	88.66%	91.98%	97
LOC	95.50%	91.86%	93.64%	393
ORG	87.88%	87.64%	87.76%	364
PER	94.19%	84.37%	89.01%	192
PRO	85.71%	57.69%	68.97%	52

SPLIT_cross-topic Zero-Shot

All experiments are done with the most successful model - XLM-RoBERTa-large (5 epochs)

3.1 Leave-one out

Train on all languages from SPLIT_cross-topic_train and SPLIT_cross-topic_test except leave_lang_id

Test on leave_lang_id from SPLIT_cross-topic_test

leave_lang_id = BG

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
BG	80.23%	84.16%	81.97%	85.58%	89.07%	87.29%	2872

Category	Precision	Recall	F1	Support
EVT	82.32%	88.82%	85.44%	304
LOC	93.08%	92.98%	93.03%	897
ORG	71.96%	88.50%	79.38%	635
PER	97.21%	90.52%	93.75%	886
PRO	56.60%	60.00%	58.25%	150

leave_lang_id = CS

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
CS	90.71%	94.40%	92.49%	93.84%	96.17%	94.99%	1567

Category	Precision	Recall	F1	Support
EVT	85.12%	91.15%	88.03%	113
LOC	97.41%	97.07%	97.24%	581
ORG	90.25%	95.25%	92.68%	379
PER	97.75%	98.74%	98.24%	396
PRO	83.02%	89.80%	86.27%	98

leave_lang_id = PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	90.71%	94.40%	92.49%	93.25%	93.87%	93.56%	2987

Category	Precision	Recall	F1	Support
EVT	91.67%	96.49%	94.02%	171
LOC	95.77%	95.50%	95.64%	1090
ORG	90.25%	89.69%	89.97%	805
PER	95.37%	96.52%	95.94%	661
PRO	87.82%	91.54%	89.64%	260

leave_lang_id = RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
RU	75.58%	78.67%	76.80%	83.21%	84.48%	83.84%	3460

Category	Precision	Recall	F1	Support
EVT	65.81%	86.78%	74.85%	295
LOC	93.50%	92.16%	92.83%	1123
ORG	78.15%	75.72%	76.91%	935
PER	91.59%	94.45%	93.00%	865
PRO	48.86%	44.21%	46.42%	242

leave_lang_id = SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
SL	80.50%	84.36%	82.05%	85.15%	89.88%	87.45%	1977

Category	Precision	Recall	F1	Support
EVT	64.85%	84.70%	74.46%	183
LOC	91.32%	94.30%	92.79%	491
ORG	81.36%	81.16%	81.26%	398
PER	90.90%	98.27%	94.44%	752
PRO	74.05%	63.40%	68.31%	153

leave_lang_id = UK

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
UK	74.60%	77.36%	74.64%	80.29%	81.24%	80.76%	1098

Category	Precision	Recall	F1	Support
EVT	77.33%	59.79%	67.44%	97
LOC	90.86%	86.01%	88.37%	393
ORG	75.21%	74.18%	74.69%	364
PER	95.96%	98.96%	97.44%	192
PRO	33.64%	69.23%	45.28%	52

3.2 One to many

Train on lang_id from SPLIT_cross-topic_train and SPLIT_cross-topic_test

Test on all except lang_id from SPLIT_cross-topic_test

lang_id = BG

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	86.80%	89.96%	88.32%	89.31%	91.20%	90.24%	2987
CS	83.46%	86.41%	84.86%	90.11%	92.41%	91.24%	1567
RU	71.32%	74.34%	72.47%	78.67%	80.14%	79.40%	3460
SL	68.26%	74.06%	70.95%	77.43%	83.97%	80.56%	1977
UK	76.19%	74.01%	72.77%	81.20%	77.50%	79.31%	1098

lang_id = CS

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	86.78%	91.38%	88.94%	89.58%	92.10%	90.82%	2987
RU	74.16%	77.57%	75.60%	80.41%	81.27%	80.84%	3460
BG	79.05%	83.76%	81.03%	85.20%	87.57%	86.37%	2872
SL	74.75%	81.58%	77.81%	81.13%	87.86%	84.36%	1977
UK	74.49%	73.91%	71.85%	79.23%	76.41%	77.79%	1098

lang_id = PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
CS	88.74%	91.72%	90.18%	91.67%	94.13%	92.88%	1567
RU	78.06%	78.22%	78.08%	83.76%	82.89%	83.32%	3460
BG	82.25%	84.26%	83.16%	86.22%	88.20%	87.19%	2872
SL	77.98%	81.34%	79.32%	83.93%	87.96%	85.90%	1977
UK	75.28%	74.04%	72.63%	80.88%	76.68%	78.73%	1098

lang_id = RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	81.73%	88.79%	84.92%	86.53%	90.56%	88.50%	2987
CS	79.87%	86.87%	83.10%	85.77%	91.58%	88.58%	1567
BG	79.07%	84.05%	81.49%	79.07%	84.05%	81.49%	2872
SL	74.04%	80.85%	76.65%	77.62%	86.65%	81.88%	1977
UK	74.65%	75.44%	73.07%	79.67%	78.87%	79.27%	1098

lang_id = SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	85.11%	90.44%	87.65%	88.21%	92.17%	90.14%	2987
CS	82.58%	86.85%	84.55%	87.73%	92.60%	90.10%	1567
RU	75.74%	77.92%	76.63%	80.31%	83.12%	81.69%	3460
BG	78.37%	83.20%	80.37%	82.43%	87.05%	84.67%	2872
UK	74.57%	72.32%	70.57%	76.24%	77.14%	76.69%	1098

lang_id = UK

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	71.39%	62.61%	60.11%	65.57%	68.53%	67.02%	2987
CS	72.33%	66.95%	64.11%	74.86%	78.49%	76.64%	1567
RU	62.59%	65.94%	61.92%	72.78%	74.65%	73.71%	3460
BG	68.03%	65.80%	64.36%	71.85%	75.00%	73.39%	2872
SL	65.81%	64.14%	58.59%	70.12%	75.27%	72.60%	1977

3.3 Many to one

Train on selected languages from SPLIT_cross-topic_train and SPLIT_cross-topic_test

Test on lang_id from SPLIT_cross-topic_test

_lang_id = BG, trained on CS + PL + SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
BG	83.42%	85.98%	84.55%	87.51%	89.21%	88.38%	2872

Category	Precision	Recall	F1	Support
EVT	85.86%	83.88%	84.86%	304
LOC	94.56%	94.87%	94.71%	897
ORG	74.21%	84.72%	79.12%	635
PER	96.42%	91.08%	93.67%	886
PRO	66.08%	75.33%	70.40%	150

_lang_id = CS, trained on BG + PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
CS	89.27%	93.33%	91.22%	92.91%	95.41%	94.14%	1567

Category	Precision	Recall	F1	Support
EVT	82.54%	92.04%	87.03%	113
LOC	96.23%	96.73%	96.48%	581
ORG	90.33%	93.67%	91.97%	379
PER	97.26%	98.48%	97.87%	396
PRO	80.00%	85.71%	82.76%	98

_lang_id = PL, trained on BG + CS + SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	90.73%	94.31%	92.41%	92.72%	93.81%	93.26%	2987

Category	Precision	Recall	F1	Support
EVT	88.30%	97.08%	92.48%	171
LOC	95.97%	96.15%	96.06%	1090
ORG	91.09%	87.58%	89.30%	805
PER	94.38%	96.52%	95.44%	661
PRO	83.90%	94.23%	88.77%	260

_lang_id = RU, trained on PL + SL + UK
leave_lang_id = RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
RU	75.56%	74.55%	74.60%	83.64%	82.72%	83.17%	3460

Category	Precision	Recall	F1	Support
EVT	67.71%	80.34%	73.49%	295
LOC	92.68%	91.27%	91.97%	1123
ORG	74.29%	76.04%	75.16%	935
PER	95.65%	94.10%	94.87%	865
PRO	47.47%	30.99%	37.50%	242

leave_lang_id = SL, trained on CS + PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
SL	79.32%	83.33%	81.17%	84.92%	88.87%	86.85%	1977

Category	Precision	Recall	F1	Support
EVT	66.82%	80.33%	72.95%	183
LOC	93.09%	93.28%	93.18%	491
ORG	76.96%	81.41%	79.12%	398
PER	92.40%	96.94%	94.61%	752
PRO	67.35%	64.71%	66.00%	153

leave_lang_id = UK, trained on BG + RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
UK	74.59%	77.24%	74.29%	79.98%	80.78%	80.38%	1098

Category	Precision	Recall	F1	Support
EVT	79.73%	60.82%	69.01%	97
LOC	91.03%	85.24%	88.04%	393
ORG	75.21%	73.35%	74.27%	364
PER	95.98%	99.48%	97.70%	192
PRO	30.97%	67.31%	42.42%	52

3.3 Many to many

Train on selected languages* from SPLIT_cross-topic_train and SPLIT_cross-topic_test

Test on all languages from SPLIT_cross-topic_test except the selected languages*

Future direction

Similar repos

tomaarsen/span-marker-mbert-base-multinerd

Model

SpanMarker for Multilingual Named Entity Recognition

sandipanadowerah/NER

Assignment: Research Engineer in Natural Language Processing (Named entity recognition)

JoelNiklaus/LEXTREME

This repository provides scripts for evaluating NLP models on the LEXTREME benchmark, a set of…

gliner-community/gliner_small-v2.5

Model

About

Babelscape/wikineural

Data and evaluation code for the paper WikiNEuRal: Combined Neural and Knowledge-based Silver Data…

Gabriel819/zeroshot_ner

Official code for Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource…

LazarusNLP/NusaBERT

NusaBERT: Teaching IndoBERT to be multilingual and multicultural!

google-research/xtreme

XTREME is a benchmark for the evaluation of the cross-lingual generalization ability of pre-trained…

650

MarioKuzmanov/Thesis

Multilingual & Cross-Lingual NER Transfer for Slavic Languages

Jupyter Notebook

7 commits

updated Oct 5, 2025

See the code

README

BA Thesis

Multilingual and Cross-Lingual NER Transfer for Slavic Languages

1. Project Setup

1.1 Setup Virtual Environment

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-*.sh
conda create -n .venv python=3.12.4

1.2 Install Requirements

pip install -r requirements.txt

1.3 Download the data

   cd data
   git clone https://github.com/SlavicNLP/SlavicNER

Start with the experiments

2. Project Structure

BA Thesis

├── data # Cross-Lingual NEs corpus

├── related_work # Related Work

├── screenshots # Confusion Matrices for models and data distribution

3. Small peek at the results - Named Entity Recognition (NER)

N.B. Please refer to the thesis for a thorough results of all methods, or get the results yourself by running scripts/eval_ner.py with the correct file paths.

SPLIT_cross-topic - Fine-Tuning

Train on all languages from SPLIT_cross-topic_train

Test on all languages from SPLIT_cross-topic_test

per-model performance on SPLIT_cross-topic_test

Support = 13961	P-macro	R-macro	F1-macro	P-weighted	R-weighted	F1-weighted	P-micro	R-micro	F1-micro
XLM-RoBERTa-large (5 epochs)	87.83%	91.31%	89.51%	91.24%	93.51%	92.36%	91.12%	93.51%	92.30%
2-BiLSTM (10 epochs)	87.11%	80.86%	83.84%	89.97%	83.14%	86.42%	90.00%	83.14%	86.43%
2-BiGRU (10 epochs)	86.28%	79.28%	82.58%	88.49%	81.97%	85.10%	88.55%	81.97%	85.13%
2-BiLSTM (5 epochs)	81.68%	81.74%	81.48%	85.67%	83.57%	84.61%	85.16%	83.57%	84.36%
2-BiGRU (5 epochs)	83.85%	78.28%	80.75%	85.46%	81.73%	83.55%	85.45%	81.73%	83.55%
2-BiSRNN (10 epochs)	80.00%	74.08%	76.85%	83.56%	79.24%	81.35%	83.86%	79.24%	81.48%
2-BiSRNN (5 epochs)	75.42%	70.29%	72.61%	80.14%	76.98%	78.53%	80.54%	76.98%	78.72%

per-category performance of XLM-RoBERTa-large (5 epochs)

Category	Precision	Recall	F1	Support
EVT	82.17%	90.71%	86.23%	1163
LOC	95.58%	96.04%	95.81%	4575
ORG	87.29%	89.65%	88.45%	3516
PER	95.72%	97.73%	96.72%	3752
PRO	78.39%	82.41%	80.35%	955

per-language performance of XLM-RoBERTa-large (5 epochs)

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	94.08%	96.29%	95.16%	95.00%	95.98%	95.49%	2987
CS	92.22%	95.95%	94.01%	94.58%	96.94%	95.75%	1567
RU	84.36%	88.78%	86.49%	88.24%	91.71%	89.94%	3460
BG	88.49%	91.65%	90.02%	92.39%	94.74%	93.55%	2872
SL	82.56%	86.79%	84.55%	87.14%	90.84%	88.95%	1977
UK	83.34%	84.53%	83.88%	88.92%	89.16%	89.04%	1098

lang_id per-category performance of XLM-RoBERTa-large (5 epochs)
lang_id = PL

Category	Precision	Recall	F1	Support
EVT	92.74%	97.08%	94.86%	171
LOC	96.98%	97.06%	97.02%	1090
ORG	92.81%	93.04%	92.93%	805
PER	96.54%	96.97%	96.75%	661
PRO	91.34%	97.31%	94.23%	260

lang_id = CS

Category	Precision	Recall	F1	Support
EVT	84.25%	94.69%	89.17%	113
LOC	97.75%	97.42%	97.59%	581
ORG	91.48%	96.31%	93.83%	379
PER	97.50%	98.48%	97.99%	396
PRO	90.10%	92.86%	91.46%	98

lang_id = RU

Category	Precision	Recall	F1	Support
EVT	82.87%	90.17%	86.36%	295
LOC	93.42%	96.08%	94.73%	1123
ORG	83.35%	86.20%	84.75%	935
PER	95.15%	97.46%	96.29%	865
PRO	67.04%	73.97%	70.33%	242

lang_id = BG

Category	Precision	Recall	F1	Support
EVT	85.24%	93.09%	88.99%	304
LOC	97.08%	96.21%	96.64%	897
ORG	87.99%	91.18%	89.56%	635
PER	96.48%	99.10%	97.77%	886
PRO	75.64%	78.67%	77.12%	150

lang_id = SL

Category	Precision	Recall	F1	Support
EVT	70.64%	84.15%	76.81%	183
LOC	93.10%	96.13%	94.59%	491
ORG	81.17%	83.42%	82.28%	398
PER	93.55%	96.41%	94.96%	752
PRO	74.34%	73.86%	74.10%	153

lang_id = UK

Category	Precision	Recall	F1	Support
EVT	73.83%	81.44%	77.45%	97
LOC	94.68%	90.59%	92.59%	393
ORG	86.52%	88.19%	87.35%	364
PER	96.94%	98.96%	97.94%	192
PRO	64.71%	63.46%	64.08%	52

per-category performance of 2-BiLSTM (10 epochs)

Category	Precision	Recall	F1	Support
EVT	88.24%	85.81%	87.01%	1163
LOC	94.77%	90.34%	92.50%	4575
ORG	86.01%	78.84%	82.27%	3516
PER	92.38%	81.45%	86.57%	3752
PRO	74.14%	67.85%	70.86%	955

per-language performance of 2-BiLSTM (10 epochs)

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	91.90%	85.85%	88.69%	92.78%	85.14%	88.79%	2987
CS	89.73%	84.76%	87.11%	92.62%	86.47%	89.44%	1567
RU	82.25%	78.71%	80.39%	85.93%	81.04%	83.42%	3460
BG	89.35%	83.02%	86.02%	91.71%	85.86%	88.69%	2872
SL	81.45%	69.64%	74.61%	86.96%	74.91%	80.49%	1977
UK	91.77%	82.04%	86.27%	92.29%	87.25%	89.70%	1098

lang_id per-category performance of 2-BiLSTM (10 epochs)
lang_id = PL

Category	Precision	Recall	F1	Support
EVT	97.56%	93.57%	95.52%	171
LOC	95.56%	88.90%	92.11%	1090
ORG	91.23%	82.73%	86.78%	805
PER	93.81%	80.18%	86.46%	661
PRO	81.34%	83.85%	82.58%	260

lang_id = CS

Category	Precision	Recall	F1	Support
EVT	86.49%	84.96%	85.71%	113
LOC	96.55%	91.39%	93.90%	581
ORG	89.43%	87.07%	88.24%	379
PER	94.96%	80.81%	87.31%	396
PRO	81.25%	79.59%	80.41%	98

lang_id = RU

Category	Precision	Recall	F1	Support
EVT	86.36%	83.73%	85.03%	295
LOC	91.71%	88.69%	90.18%	1123
ORG	83.41%	74.76%	78.85%	935
PER	88.46%	81.50%	84.84%	865
PRO	61.33%	64.88%	63.05%	242

lang_id = BG

Category	Precision	Recall	F1	Support
EVT	88.89%	86.84%	87.85%	304
LOC	97.00%	93.76%	95.35%	897
ORG	86.15%	80.31%	83.13%	635
PER	92.67%	84.20%	88.23%	886
PRO	82.03%	70.00%	75.54%	150

lang_id = SL

Category	Precision	Recall	F1	Support
EVT	79.23%	79.23%	79.23%	183
LOC	93.35%	88.59%	90.91%	491
ORG	74.92%	62.31%	68.04%	398
PER	93.83%	78.86%	85.69%	752
PRO	65.93%	39.22%	49.18%	153

lang_id = UK

Category	Precision	Recall	F1	Support
EVT	95.56%	88.66%	91.98%	97
LOC	95.50%	91.86%	93.64%	393
ORG	87.88%	87.64%	87.76%	364
PER	94.19%	84.37%	89.01%	192
PRO	85.71%	57.69%	68.97%	52

SPLIT_cross-topic Zero-Shot

All experiments are done with the most successful model - XLM-RoBERTa-large (5 epochs)

3.1 Leave-one out

Train on all languages from SPLIT_cross-topic_train and SPLIT_cross-topic_test except leave_lang_id

Test on leave_lang_id from SPLIT_cross-topic_test

leave_lang_id = BG

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
BG	80.23%	84.16%	81.97%	85.58%	89.07%	87.29%	2872

Category	Precision	Recall	F1	Support
EVT	82.32%	88.82%	85.44%	304
LOC	93.08%	92.98%	93.03%	897
ORG	71.96%	88.50%	79.38%	635
PER	97.21%	90.52%	93.75%	886
PRO	56.60%	60.00%	58.25%	150

leave_lang_id = CS

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
CS	90.71%	94.40%	92.49%	93.84%	96.17%	94.99%	1567

Category	Precision	Recall	F1	Support
EVT	85.12%	91.15%	88.03%	113
LOC	97.41%	97.07%	97.24%	581
ORG	90.25%	95.25%	92.68%	379
PER	97.75%	98.74%	98.24%	396
PRO	83.02%	89.80%	86.27%	98

leave_lang_id = PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	90.71%	94.40%	92.49%	93.25%	93.87%	93.56%	2987

Category	Precision	Recall	F1	Support
EVT	91.67%	96.49%	94.02%	171
LOC	95.77%	95.50%	95.64%	1090
ORG	90.25%	89.69%	89.97%	805
PER	95.37%	96.52%	95.94%	661
PRO	87.82%	91.54%	89.64%	260

leave_lang_id = RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
RU	75.58%	78.67%	76.80%	83.21%	84.48%	83.84%	3460

Category	Precision	Recall	F1	Support
EVT	65.81%	86.78%	74.85%	295
LOC	93.50%	92.16%	92.83%	1123
ORG	78.15%	75.72%	76.91%	935
PER	91.59%	94.45%	93.00%	865
PRO	48.86%	44.21%	46.42%	242

leave_lang_id = SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
SL	80.50%	84.36%	82.05%	85.15%	89.88%	87.45%	1977

Category	Precision	Recall	F1	Support
EVT	64.85%	84.70%	74.46%	183
LOC	91.32%	94.30%	92.79%	491
ORG	81.36%	81.16%	81.26%	398
PER	90.90%	98.27%	94.44%	752
PRO	74.05%	63.40%	68.31%	153

leave_lang_id = UK

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
UK	74.60%	77.36%	74.64%	80.29%	81.24%	80.76%	1098

Category	Precision	Recall	F1	Support
EVT	77.33%	59.79%	67.44%	97
LOC	90.86%	86.01%	88.37%	393
ORG	75.21%	74.18%	74.69%	364
PER	95.96%	98.96%	97.44%	192
PRO	33.64%	69.23%	45.28%	52

3.2 One to many

Train on lang_id from SPLIT_cross-topic_train and SPLIT_cross-topic_test

Test on all except lang_id from SPLIT_cross-topic_test

lang_id = BG

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	86.80%	89.96%	88.32%	89.31%	91.20%	90.24%	2987
CS	83.46%	86.41%	84.86%	90.11%	92.41%	91.24%	1567
RU	71.32%	74.34%	72.47%	78.67%	80.14%	79.40%	3460
SL	68.26%	74.06%	70.95%	77.43%	83.97%	80.56%	1977
UK	76.19%	74.01%	72.77%	81.20%	77.50%	79.31%	1098

lang_id = CS

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	86.78%	91.38%	88.94%	89.58%	92.10%	90.82%	2987
RU	74.16%	77.57%	75.60%	80.41%	81.27%	80.84%	3460
BG	79.05%	83.76%	81.03%	85.20%	87.57%	86.37%	2872
SL	74.75%	81.58%	77.81%	81.13%	87.86%	84.36%	1977
UK	74.49%	73.91%	71.85%	79.23%	76.41%	77.79%	1098

lang_id = PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
CS	88.74%	91.72%	90.18%	91.67%	94.13%	92.88%	1567
RU	78.06%	78.22%	78.08%	83.76%	82.89%	83.32%	3460
BG	82.25%	84.26%	83.16%	86.22%	88.20%	87.19%	2872
SL	77.98%	81.34%	79.32%	83.93%	87.96%	85.90%	1977
UK	75.28%	74.04%	72.63%	80.88%	76.68%	78.73%	1098

lang_id = RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	81.73%	88.79%	84.92%	86.53%	90.56%	88.50%	2987
CS	79.87%	86.87%	83.10%	85.77%	91.58%	88.58%	1567
BG	79.07%	84.05%	81.49%	79.07%	84.05%	81.49%	2872
SL	74.04%	80.85%	76.65%	77.62%	86.65%	81.88%	1977
UK	74.65%	75.44%	73.07%	79.67%	78.87%	79.27%	1098

lang_id = SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	85.11%	90.44%	87.65%	88.21%	92.17%	90.14%	2987
CS	82.58%	86.85%	84.55%	87.73%	92.60%	90.10%	1567
RU	75.74%	77.92%	76.63%	80.31%	83.12%	81.69%	3460
BG	78.37%	83.20%	80.37%	82.43%	87.05%	84.67%	2872
UK	74.57%	72.32%	70.57%	76.24%	77.14%	76.69%	1098

lang_id = UK

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	71.39%	62.61%	60.11%	65.57%	68.53%	67.02%	2987
CS	72.33%	66.95%	64.11%	74.86%	78.49%	76.64%	1567
RU	62.59%	65.94%	61.92%	72.78%	74.65%	73.71%	3460
BG	68.03%	65.80%	64.36%	71.85%	75.00%	73.39%	2872
SL	65.81%	64.14%	58.59%	70.12%	75.27%	72.60%	1977

3.3 Many to one

Train on selected languages from SPLIT_cross-topic_train and SPLIT_cross-topic_test

Test on lang_id from SPLIT_cross-topic_test

_lang_id = BG, trained on CS + PL + SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
BG	83.42%	85.98%	84.55%	87.51%	89.21%	88.38%	2872

Category	Precision	Recall	F1	Support
EVT	85.86%	83.88%	84.86%	304
LOC	94.56%	94.87%	94.71%	897
ORG	74.21%	84.72%	79.12%	635
PER	96.42%	91.08%	93.67%	886
PRO	66.08%	75.33%	70.40%	150

_lang_id = CS, trained on BG + PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
CS	89.27%	93.33%	91.22%	92.91%	95.41%	94.14%	1567

Category	Precision	Recall	F1	Support
EVT	82.54%	92.04%	87.03%	113
LOC	96.23%	96.73%	96.48%	581
ORG	90.33%	93.67%	91.97%	379
PER	97.26%	98.48%	97.87%	396
PRO	80.00%	85.71%	82.76%	98

_lang_id = PL, trained on BG + CS + SL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
PL	90.73%	94.31%	92.41%	92.72%	93.81%	93.26%	2987

Category	Precision	Recall	F1	Support
EVT	88.30%	97.08%	92.48%	171
LOC	95.97%	96.15%	96.06%	1090
ORG	91.09%	87.58%	89.30%	805
PER	94.38%	96.52%	95.44%	661
PRO	83.90%	94.23%	88.77%	260

_lang_id = RU, trained on PL + SL + UK
leave_lang_id = RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
RU	75.56%	74.55%	74.60%	83.64%	82.72%	83.17%	3460

Category	Precision	Recall	F1	Support
EVT	67.71%	80.34%	73.49%	295
LOC	92.68%	91.27%	91.97%	1123
ORG	74.29%	76.04%	75.16%	935
PER	95.65%	94.10%	94.87%	865
PRO	47.47%	30.99%	37.50%	242

leave_lang_id = SL, trained on CS + PL

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
SL	79.32%	83.33%	81.17%	84.92%	88.87%	86.85%	1977

Category	Precision	Recall	F1	Support
EVT	66.82%	80.33%	72.95%	183
LOC	93.09%	93.28%	93.18%	491
ORG	76.96%	81.41%	79.12%	398
PER	92.40%	96.94%	94.61%	752
PRO	67.35%	64.71%	66.00%	153

leave_lang_id = UK, trained on BG + RU

Language	P-macro	R-macro	F1-macro	P-micro	R-micro	F1-micro	Support
UK	74.59%	77.24%	74.29%	79.98%	80.78%	80.38%	1098

Category	Precision	Recall	F1	Support
EVT	79.73%	60.82%	69.01%	97
LOC	91.03%	85.24%	88.04%	393
ORG	75.21%	73.35%	74.27%	364
PER	95.98%	99.48%	97.70%	192
PRO	30.97%	67.31%	42.42%	52

3.3 Many to many

Train on selected languages* from SPLIT_cross-topic_train and SPLIT_cross-topic_test

Test on all languages from SPLIT_cross-topic_test except the selected languages*

Future direction

Similar repos

tomaarsen/span-marker-mbert-base-multinerd

Model

SpanMarker for Multilingual Named Entity Recognition

sandipanadowerah/NER

Assignment: Research Engineer in Natural Language Processing (Named entity recognition)

JoelNiklaus/LEXTREME

This repository provides scripts for evaluating NLP models on the LEXTREME benchmark, a set of…

gliner-community/gliner_small-v2.5

Model

About

Babelscape/wikineural

Data and evaluation code for the paper WikiNEuRal: Combined Neural and Knowledge-based Silver Data…

Gabriel819/zeroshot_ner

Official code for Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource…

LazarusNLP/NusaBERT

NusaBERT: Teaching IndoBERT to be multilingual and multicultural!

google-research/xtreme

XTREME is a benchmark for the evaluation of the cross-lingual generalization ability of pre-trained…

650