Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark

AutoRAG example about benchmarking Korean embeddings.

Python

13 commits

updated Oct 2, 2024

See the code

README

Korean Embedding Benchmark with AutoRAG

This is a benchmark of Korean embedding models. With AutoRAG, you can make this kind of benchmark easy and fast.

Result

Top-k 1

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.3596	0.3596	0.3596	0.3596	0.3596	0.3596
KoSimCSE-roberta	0.4298	0.4298	0.4298	0.4298	0.4298	0.4298
Cohere embed-multilingual-v3.0	0.3596	0.3596	0.3596	0.3596	0.3596	0.3596
openai ada 002	0.4737	0.4737	0.4737	0.4737	0.4737	0.4737
multilingual-e5-large-instruct	0.4649	0.4649	0.4649	0.4649	0.4649	0.4649
Upstage Embedding	0.6579	0.6579	0.6579	0.6579	0.6579	0.6579
paraphrase-multilingual-MiniLM-L12-v2	0.2982	0.2982	0.2982	0.2982	0.2982	0.2982
openai_embed_3_small	0.5439	0.5439	0.5439	0.5439	0.5439	0.5439
ko-sroberta-multitask	0.4211	0.4211	0.4211	0.4211	0.4211	0.4211
openai_embed_3_large	0.6053	0.6053	0.6053	0.6053	0.6053	0.6053
KU-HIAI-ONTHEIT-large-v1	0.7105	0.7105	0.7105	0.7105	0.7105	0.7105
KU-HIAI-ONTHEIT-large-v1.1	0.7193	0.7193	0.7193	0.7193	0.7193	0.7193
kf-deberta-multitask	0.4561	0.4561	0.4561	0.4561	0.4561	0.4561
gte-multilingual-base	0.5877	0.5877	0.5877	0.5877	0.5877	0.5877
bge-m3	0.6754	0.6754	0.6754	0.6754	0.6754	0.6754
KoE5	0.6930	0.6930	0.6930	0.6930	0.6930	0.6930

Top-k 3

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.2368	0.4737	0.1579	0.2032	0.2032	0.2712
KoSimCSE-roberta	0.3026	0.6053	0.2018	0.2661	0.2661	0.3515
Cohere embed-multilingual-v3.0	0.2851	0.5702	0.1901	0.2515	0.2515	0.3321
openai ada 002	0.3553	0.7105	0.2368	0.3202	0.3202	0.4186
multilingual-e5-large-instruct	0.3333	0.6667	0.2222	0.2909	0.2909	0.3856
Upstage Embedding	0.4211	0.8421	0.2807	0.3509	0.3509	0.4743
paraphrase-multilingual-MiniLM-L12-v2	0.2061	0.4123	0.1374	0.1740	0.1740	0.2340
openai_embed_3_small	0.3640	0.7281	0.2427	0.3026	0.3026	0.4097
ko-sroberta-multitask	0.2939	0.5877	0.1959	0.2500	0.2500	0.3351
openai_embed_3_large	0.3947	0.7895	0.2632	0.3348	0.3348	0.4491
KU-HIAI-ONTHEIT-large-v1	0.4386	0.8772	0.2924	0.3421	0.3421	0.4766
KU-HIAI-ONTHEIT-large-v1.1	0.4430	0.8860	0.2953	0.3406	0.3406	0.4778
kf-deberta-multitask	0.3158	0.6316	0.2105	0.2792	0.2792	0.3679
gte-multilingual-base	0.4035	0.8070	0.2690	0.3450	0.3450	0.4614
bge-m3	0.4342	0.8684	0.2895	0.3436	0.3436	0.4757
KoE5	0.4386	0.8772	0.2924	0.3406	0.3406	0.4757

Top-k 5

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.1813	0.5439	0.1088	0.1575	0.1575	0.2491
KoSimCSE-roberta	0.2164	0.6491	0.1298	0.1751	0.1751	0.2873
Cohere embed-multilingual-v3.0	0.2076	0.6228	0.1246	0.1640	0.1640	0.2731
openai ada 002	0.2602	0.7807	0.1561	0.2139	0.2139	0.3486
multilingual-e5-large-instruct	0.2544	0.7632	0.1526	0.2194	0.2194	0.3487
Upstage Embedding	0.2982	0.8947	0.1789	0.2237	0.2237	0.3822
paraphrase-multilingual-MiniLM-L12-v2	0.1637	0.4912	0.0982	0.1437	0.1437	0.2264
openai_embed_3_small	0.2690	0.8070	0.1614	0.2148	0.2148	0.3553
ko-sroberta-multitask	0.2164	0.6491	0.1298	0.1697	0.1697	0.2835
openai_embed_3_large	0.2807	0.8421	0.1684	0.2088	0.2088	0.3586
KU-HIAI-ONTHEIT-large-v1	0.3041	0.9123	0.1825	0.2137	0.2137	0.3783
KU-HIAI-ONTHEIT-large-v1.1	0.3099	0.9298	0.1860	0.2148	0.2148	0.3834
kf-deberta-multitask	0.2281	0.6842	0.1368	0.1724	0.1724	0.2939
gte-multilingual-base	0.2865	0.8596	0.1719	0.2096	0.2096	0.3637
bge-m3	0.3099	0.9298	0.1860	0.2221	0.2221	0.3894
KoE5	0.3012	0.9035	0.1807	0.2051	0.2051	0.3697

Top-k 10

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.1212	0.6667	0.0667	0.1197	0.1197	0.2382
KoSimCSE-roberta	0.1324	0.7281	0.0728	0.1080	0.1080	0.2411
Cohere embed-multilingual-v3.0	0.1324	0.7281	0.0728	0.1150	0.1150	0.2473
openai ada 002	0.1563	0.8596	0.0860	0.1051	0.1051	0.2673
multilingual-e5-large-instruct	0.1483	0.8158	0.0816	0.0980	0.0980	0.2520
Upstage Embedding	0.1707	0.9386	0.0939	0.1078	0.1078	0.2848
paraphrase-multilingual-MiniLM-L12-v2	0.1053	0.5789	0.0579	0.0961	0.0961	0.2006
openai_embed_3_small	0.1547	0.8509	0.0851	0.0984	0.0984	0.2593
ko-sroberta-multitask	0.1276	0.7018	0.0702	0.0986	0.0986	0.2275
openai_embed_3_large	0.1643	0.9035	0.0904	0.1180	0.1180	0.2855
KU-HIAI-ONTHEIT-large-v1	0.1707	0.9386	0.0939	0.1105	0.1105	0.2860
KU-HIAI-ONTHEIT-large-v1.1	0.1722	0.9474	0.0947	0.1033	0.1033	0.2822
kf-deberta-multitask	0.1388	0.7632	0.0763	0.1	0.1	0.2422
gte-multilingual-base	0.1675	0.9211	0.0921	0.1066	0.1066	0.2805
bge-m3	0.1754	0.9649	0.0965	0.1125	0.1125	0.2939
KoE5	0.1675	0.9211	0.0921	0.0993	0.0993	0.2734

Top-k 50

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.0320	0.8158	0.0163	0.0233	0.0233	0.1529
KoSimCSE-roberta	0.0368	0.9386	0.0188	0.0270	0.0270	0.1758
Cohere embed-multilingual-v3.0	0.0382	0.9737	0.0195	0.0220	0.0220	0.1763
openai ada 002	0.0375	0.9561	0.0191	0.0295	0.0295	0.1789
multilingual-e5-large-instruct	0.0378	0.9649	0.0193	0.0295	0.0295	0.1804
Upstage Embedding	0.0392	1.0000	0.0200	0.0206	0.0206	0.1776
paraphrase-multilingual-MiniLM-L12-v2	0.0313	0.7982	0.0160	0.0218	0.0218	0.1503
openai_embed_3_small	0.0382	0.9737	0.0195	0.0202	0.0202	0.1731
ko-sroberta-multitask	0.0354	0.9035	0.0181	0.0245	0.0245	0.1691
openai_embed_3_large	0.0382	0.9737	0.0195	0.0210	0.0210	0.1741
KU-HIAI-ONTHEIT-large-v1	0.0385	0.9825	0.0196	0.0212	0.0212	0.1758
KU-HIAI-ONTHEIT-large-v1.1	0.0385	0.9825	0.0196	0.0206	0.0206	0.1750
kf-deberta-multitask	0.0351	0.8947	0.0179	0.0228	0.0228	0.1654
gte-multilingual-base	0.0392	1.0000	0.0200	0.0259	0.0259	0.1834
bge-m3	0.0392	1.0000	0.0200	0.0206	0.0206	0.1775
KoE5	0.0385	0.9824	0.0196	0.0208	0.0208	0.1752

Installation

pip install -r requirements.txt

Running the project

Make .env file using .env.template file. You have to prepare three api keys, openai, cohere, and upstage.
Run evaluator with the following command.

python main.py --project_dir ./project_dir

Check the result in the project_dir folder.

Significant stargazers

Jiwook Han

19 followers · starred Oct 2024

Similar repos

nlpai-lab/KURE-v1

Model

🔎 KURE-v1

nlpai-lab/KoE5

Model

🔎 KoE5

dragonkue/bge-reranker-v2-m3-ko

Model

Reranker (Cross-Encoder)

instructkr/reranker-simple-benchmark

Make running benchmark simple yet maintainable, again. Now only supports Korean-based cross-encoder.

nlpai-lab/KURE

🔎 KURE: Korea University Retrieval Embedding models

242

HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5

Model

KaLM-Embedding is a series of embedding models adapted from auto-regressive LLMs with superior…

BM-K/Korean-MTEB-Retrieval-Evaluators

Korean MTEB Retrieval Evaluators for SPLADE, Dense, and Reranking models

nlpai-lab/KURE-Reranker-base

Model

KURE-Reranker-base

Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark

AutoRAG example about benchmarking Korean embeddings.

Python

13 commits

updated Oct 2, 2024

See the code

README

Korean Embedding Benchmark with AutoRAG

This is a benchmark of Korean embedding models. With AutoRAG, you can make this kind of benchmark easy and fast.

Result

Top-k 1

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.3596	0.3596	0.3596	0.3596	0.3596	0.3596
KoSimCSE-roberta	0.4298	0.4298	0.4298	0.4298	0.4298	0.4298
Cohere embed-multilingual-v3.0	0.3596	0.3596	0.3596	0.3596	0.3596	0.3596
openai ada 002	0.4737	0.4737	0.4737	0.4737	0.4737	0.4737
multilingual-e5-large-instruct	0.4649	0.4649	0.4649	0.4649	0.4649	0.4649
Upstage Embedding	0.6579	0.6579	0.6579	0.6579	0.6579	0.6579
paraphrase-multilingual-MiniLM-L12-v2	0.2982	0.2982	0.2982	0.2982	0.2982	0.2982
openai_embed_3_small	0.5439	0.5439	0.5439	0.5439	0.5439	0.5439
ko-sroberta-multitask	0.4211	0.4211	0.4211	0.4211	0.4211	0.4211
openai_embed_3_large	0.6053	0.6053	0.6053	0.6053	0.6053	0.6053
KU-HIAI-ONTHEIT-large-v1	0.7105	0.7105	0.7105	0.7105	0.7105	0.7105
KU-HIAI-ONTHEIT-large-v1.1	0.7193	0.7193	0.7193	0.7193	0.7193	0.7193
kf-deberta-multitask	0.4561	0.4561	0.4561	0.4561	0.4561	0.4561
gte-multilingual-base	0.5877	0.5877	0.5877	0.5877	0.5877	0.5877
bge-m3	0.6754	0.6754	0.6754	0.6754	0.6754	0.6754
KoE5	0.6930	0.6930	0.6930	0.6930	0.6930	0.6930

Top-k 3

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.2368	0.4737	0.1579	0.2032	0.2032	0.2712
KoSimCSE-roberta	0.3026	0.6053	0.2018	0.2661	0.2661	0.3515
Cohere embed-multilingual-v3.0	0.2851	0.5702	0.1901	0.2515	0.2515	0.3321
openai ada 002	0.3553	0.7105	0.2368	0.3202	0.3202	0.4186
multilingual-e5-large-instruct	0.3333	0.6667	0.2222	0.2909	0.2909	0.3856
Upstage Embedding	0.4211	0.8421	0.2807	0.3509	0.3509	0.4743
paraphrase-multilingual-MiniLM-L12-v2	0.2061	0.4123	0.1374	0.1740	0.1740	0.2340
openai_embed_3_small	0.3640	0.7281	0.2427	0.3026	0.3026	0.4097
ko-sroberta-multitask	0.2939	0.5877	0.1959	0.2500	0.2500	0.3351
openai_embed_3_large	0.3947	0.7895	0.2632	0.3348	0.3348	0.4491
KU-HIAI-ONTHEIT-large-v1	0.4386	0.8772	0.2924	0.3421	0.3421	0.4766
KU-HIAI-ONTHEIT-large-v1.1	0.4430	0.8860	0.2953	0.3406	0.3406	0.4778
kf-deberta-multitask	0.3158	0.6316	0.2105	0.2792	0.2792	0.3679
gte-multilingual-base	0.4035	0.8070	0.2690	0.3450	0.3450	0.4614
bge-m3	0.4342	0.8684	0.2895	0.3436	0.3436	0.4757
KoE5	0.4386	0.8772	0.2924	0.3406	0.3406	0.4757

Top-k 5

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.1813	0.5439	0.1088	0.1575	0.1575	0.2491
KoSimCSE-roberta	0.2164	0.6491	0.1298	0.1751	0.1751	0.2873
Cohere embed-multilingual-v3.0	0.2076	0.6228	0.1246	0.1640	0.1640	0.2731
openai ada 002	0.2602	0.7807	0.1561	0.2139	0.2139	0.3486
multilingual-e5-large-instruct	0.2544	0.7632	0.1526	0.2194	0.2194	0.3487
Upstage Embedding	0.2982	0.8947	0.1789	0.2237	0.2237	0.3822
paraphrase-multilingual-MiniLM-L12-v2	0.1637	0.4912	0.0982	0.1437	0.1437	0.2264
openai_embed_3_small	0.2690	0.8070	0.1614	0.2148	0.2148	0.3553
ko-sroberta-multitask	0.2164	0.6491	0.1298	0.1697	0.1697	0.2835
openai_embed_3_large	0.2807	0.8421	0.1684	0.2088	0.2088	0.3586
KU-HIAI-ONTHEIT-large-v1	0.3041	0.9123	0.1825	0.2137	0.2137	0.3783
KU-HIAI-ONTHEIT-large-v1.1	0.3099	0.9298	0.1860	0.2148	0.2148	0.3834
kf-deberta-multitask	0.2281	0.6842	0.1368	0.1724	0.1724	0.2939
gte-multilingual-base	0.2865	0.8596	0.1719	0.2096	0.2096	0.3637
bge-m3	0.3099	0.9298	0.1860	0.2221	0.2221	0.3894
KoE5	0.3012	0.9035	0.1807	0.2051	0.2051	0.3697

Top-k 10

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.1212	0.6667	0.0667	0.1197	0.1197	0.2382
KoSimCSE-roberta	0.1324	0.7281	0.0728	0.1080	0.1080	0.2411
Cohere embed-multilingual-v3.0	0.1324	0.7281	0.0728	0.1150	0.1150	0.2473
openai ada 002	0.1563	0.8596	0.0860	0.1051	0.1051	0.2673
multilingual-e5-large-instruct	0.1483	0.8158	0.0816	0.0980	0.0980	0.2520
Upstage Embedding	0.1707	0.9386	0.0939	0.1078	0.1078	0.2848
paraphrase-multilingual-MiniLM-L12-v2	0.1053	0.5789	0.0579	0.0961	0.0961	0.2006
openai_embed_3_small	0.1547	0.8509	0.0851	0.0984	0.0984	0.2593
ko-sroberta-multitask	0.1276	0.7018	0.0702	0.0986	0.0986	0.2275
openai_embed_3_large	0.1643	0.9035	0.0904	0.1180	0.1180	0.2855
KU-HIAI-ONTHEIT-large-v1	0.1707	0.9386	0.0939	0.1105	0.1105	0.2860
KU-HIAI-ONTHEIT-large-v1.1	0.1722	0.9474	0.0947	0.1033	0.1033	0.2822
kf-deberta-multitask	0.1388	0.7632	0.0763	0.1	0.1	0.2422
gte-multilingual-base	0.1675	0.9211	0.0921	0.1066	0.1066	0.2805
bge-m3	0.1754	0.9649	0.0965	0.1125	0.1125	0.2939
KoE5	0.1675	0.9211	0.0921	0.0993	0.0993	0.2734

Top-k 50

Model name	F1	Recall	Precision	mAP	mRR	NDCG
paraphrase-multilingual-mpnet-base-v2	0.0320	0.8158	0.0163	0.0233	0.0233	0.1529
KoSimCSE-roberta	0.0368	0.9386	0.0188	0.0270	0.0270	0.1758
Cohere embed-multilingual-v3.0	0.0382	0.9737	0.0195	0.0220	0.0220	0.1763
openai ada 002	0.0375	0.9561	0.0191	0.0295	0.0295	0.1789
multilingual-e5-large-instruct	0.0378	0.9649	0.0193	0.0295	0.0295	0.1804
Upstage Embedding	0.0392	1.0000	0.0200	0.0206	0.0206	0.1776
paraphrase-multilingual-MiniLM-L12-v2	0.0313	0.7982	0.0160	0.0218	0.0218	0.1503
openai_embed_3_small	0.0382	0.9737	0.0195	0.0202	0.0202	0.1731
ko-sroberta-multitask	0.0354	0.9035	0.0181	0.0245	0.0245	0.1691
openai_embed_3_large	0.0382	0.9737	0.0195	0.0210	0.0210	0.1741
KU-HIAI-ONTHEIT-large-v1	0.0385	0.9825	0.0196	0.0212	0.0212	0.1758
KU-HIAI-ONTHEIT-large-v1.1	0.0385	0.9825	0.0196	0.0206	0.0206	0.1750
kf-deberta-multitask	0.0351	0.8947	0.0179	0.0228	0.0228	0.1654
gte-multilingual-base	0.0392	1.0000	0.0200	0.0259	0.0259	0.1834
bge-m3	0.0392	1.0000	0.0200	0.0206	0.0206	0.1775
KoE5	0.0385	0.9824	0.0196	0.0208	0.0208	0.1752

Installation

pip install -r requirements.txt

Running the project

Make .env file using .env.template file. You have to prepare three api keys, openai, cohere, and upstage.
Run evaluator with the following command.

python main.py --project_dir ./project_dir

Check the result in the project_dir folder.

Significant stargazers

Jiwook Han

19 followers · starred Oct 2024

Similar repos

nlpai-lab/KURE-v1

Model

🔎 KURE-v1

nlpai-lab/KoE5

Model

🔎 KoE5

dragonkue/bge-reranker-v2-m3-ko

Model

Reranker (Cross-Encoder)

instructkr/reranker-simple-benchmark

Make running benchmark simple yet maintainable, again. Now only supports Korean-based cross-encoder.

nlpai-lab/KURE

🔎 KURE: Korea University Retrieval Embedding models

242

HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5

Model

KaLM-Embedding is a series of embedding models adapted from auto-regressive LLMs with superior…

BM-K/Korean-MTEB-Retrieval-Evaluators

Korean MTEB Retrieval Evaluators for SPLADE, Dense, and Reranking models

nlpai-lab/KURE-Reranker-base

Model

KURE-Reranker-base