Wrapper around the google SentencePiece tokenizer. Used to tokenize text for language models and other NLP tasks.
C#
1
21 commits
updated Mar 10, 2026
README
SentencePiece Tokenizer
Usage
The tokenizers should be thread-safe, as the underlying sentencepiece processor is thread-safe. This has not been extensively tested however!
Marian Tokenizer
Uses the tokenizer of a sentencepiece model, but a different vocabulary for the ids.
XlmRoberta Tokenizer
Use the facebook xlm-roberta-base (Alt1, Alt2) model
References
- Using SentencePiece v0.2.0 from 2024-02-19
- For BERT-style embeddings it is recommended to use FastBertTokenizer
- Inspired by SIL.Machine.Tokenization.SentencePiece
Wrapper around the google SentencePiece tokenizer. Used to tokenize text for language models and other NLP tasks.
C#
1
21 commits
updated Mar 10, 2026
README
SentencePiece Tokenizer
Usage
The tokenizers should be thread-safe, as the underlying sentencepiece processor is thread-safe. This has not been extensively tested however!
Marian Tokenizer
Uses the tokenizer of a sentencepiece model, but a different vocabulary for the ids.
XlmRoberta Tokenizer
Use the facebook xlm-roberta-base (Alt1, Alt2) model
References
- Using SentencePiece v0.2.0 from 2024-02-19
- For BERT-style embeddings it is recommended to use FastBertTokenizer
- Inspired by SIL.Machine.Tokenization.SentencePiece