Thai-Sentence-Vector-Benchmark

Benchmark for Thai sentence representation based on Thai STS-B, Text classification, and Retrieval datasets.

Motivation

Sentence representation plays a crucial role in NLP downstream tasks such as NLI, text classification, and STS. Recent sentence representation training techniques require NLI or STS datasets. However, no equivalent Thai NLI or STS datasets exist for sentence representation training. To address this problem, we create "Thai sentence vector benchmark" to demonstrate that we can train Thai sentence representation without any supervised dataset.

Our first preliminary results demonstrate that we can train a robust sentence representation model with an unsupervised technique called SimCSE. We show that it is possible to train SimCSE with 1.3 M sentences from Wikipedia within 2 hours on the Google Colab (V100), where the performance of SimCSE-XLM-R is similar to mDistil-BERT<-mUSE (train on > 1B sentences).

Moreover, we provide the Thai sentence vector benchmark. Our benchmark aims to evaluate the effectiveness of sentence embedding models on Thai zero-shot and transfer learning tasks. The tasks comprise of four tasks: Semantic ranking on STS-B, text classification (transfer), pair classification, and retrieval question answering (QA).

Install

conda create -n thai_sentence_vector_benchmark python==3.11.4
conda activate thai_sentence_vector_benchmark

# Select the appropriate PyTorch version based on your CUDA version
# CUDA 11.8
conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 pytorch-cuda=11.8 -c pytorch -c nvidia
# CUDA 12.1
conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 pytorch-cuda=12.1 -c pytorch -c nvidia
# CPU Only
conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 cpuonly -c pytorch

pip install -e .

Reproduce the results

python scripts/eval_all.py \
--cohere_api_key <YOUR_COHERE_API_KEY> \
--openai_api_key <YOUR_OPENAI_API_KEY>

Usage

from sentence_transformers import SentenceTransformer
from thai_sentence_vector_benchmark.benchmark import ThaiSentenceVectorBenchmark

model = SentenceTransformer("intfloat/e5-mistral-7b-instruct")
benchmark = ThaiSentenceVectorBenchmark()
results = benchmark(
  model,
  task_prompts={
    "sts": "Instruct: Retrieve semantically similar text.\nQuery: ",
    "retrieval": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: ",
    "pair_classification": "Instruct: Retrieve parallel sentences.\nQuery: ",
    "text_classification": "Instruct: Classify the sentiment of the text.\nText: ",
  },
  batch_size=1, 
)
>> {
  "STS": {
    "sts_b": {"Spearman_Correlation": float},
    "Average": {"Spearman_Correlation": float},
  },
  "Text_Classification": {
    "wisesight": {"Accuracy": float, "F1": float},
    "wongnai": {"Accuracy": float, "F1": float},
    "generated_reviews": {"Accuracy": float, "F1": float},
    "Average": {"Accuracy": float, "F1": float},
  },
  "Pair_Classification": {
    "xnli": {"AP": float},
    "Average": {"AP": float},
  },
  "Retrieval": {
    "xquad": {"R@1": float, "MRR@10": float},
    "miracl": {"R@1": float, "MRR@10": float},
    "tydiqa": {"R@1": float, "MRR@10": float},
    "Average": {"R@1": float, "MRR@10": float},
  },
  "Average": float,
}

How do we train unsupervised sentence representation?

We provide simple and effective sentence embedding methods that do not require supervised labels (unsupervised learning) as follows:

SimCSE

• We use SimCSE:Simple Contrastive Learning of Sentence Embeddings on multilingual LM models (mBERT, distil-mBERT, XLM-R) and a monolingual model (WangchanBERTa).
• Training data: Thai Wikipedia.
• Example: SimCSE-Thai.ipynb.
• Training Example on Google Colab: https://colab.research.google.com/github/mrpeerat/Thai-Sentence-Vector-Benchmark/blob/main/SimCSE-Thai.ipynb

ConGen

• We use the training objective from ConGen on various PLMs.
• Training data: scb-mt-en-th-2020
• Example: ConGen-Thai.ipynb

SCT

• We use the training objective from SCT on various PLMs.
• Training data: scb-mt-en-th-2020
• Example: SCT-Thai.ipynb

Why do we select these techniques?

• Easy to train
• Compatible with every model
• Do not require any annotated dataset
• The best sentence representation method (for now) in terms of the performance on STS and downstream tasks (SCT outperformed ConGen and SimCSE in their paper).

What about other techniques?

We also consider other techniques (supervised and unsupervised methods) in this repository. Currently, we have various methods tested on our benchmarks, such as:

• Supervised learning: sentence-bert.
• Multilingual sentence representation alignment: CL-ReLKT (NAACL'22)

Thai semantic textual similarity benchmark

• We use STS-B translated ver. in which we translate STS-B from SentEval by using google-translate API
• How to evaluate sentence representation: Easy_Evaluation.ipynb
• How to evaluate sentence representation on Google Colab: https://colab.research.google.com/github/mrpeerat/Thai-Sentence-Vector-Benchmark/blob/main/SentEval.ipynb

Thai transfer benchmark

• We use Wisesight, Wongnai, and Generated review datasets.
• How to evaluate: Transfer_Evaluation

Wisesight

Wongnai

Generated Review

Thai pair classification benchmark

• We use XNLI dev and test set. We drop neutral classes and change from contradiction => 0 and entailment =>1.
• We use the average precision score as the main metric.
• How to evaluate: XNLI_evaluation.ipynb

Thai retrieval benchmark

• We use XQuAD, MIRACL, and TyDiQA datasets.
• How to evaluate: Retrieval_Evaluation

XQuAD

MIRACL

TyDiQA

Thank you for the many codes from

• Sentence-transformer (Sentence-BERT)
• SimCSE github
• ConGen github
• SCT github

Acknowledgments:

• Can: proofread
• Charin: proofread + idea