Shadowgraph
About
Sign in

r4ghu/tokenizers.js

Port of the lightweight tokenizers.js from huggingface/transformers.js

0

stars

8

commits

JavaScript

primary language

Oct 20, 2024

updated

README

tokenizers.js

Port of the lightweight tokenizers.js from huggingface/transformers.js

Goal

  • Create a light weight tokenizers.js from @huggingface/transformers.js
  • Make transformers.js utilize this tokenizers.js
  • (If possible) transfer the project repo to Hugging Face. For example, @huggingface/tokenizers.js

Contributors

r4ghu

8 commits

Similar repos

lenML/tokenizers

a lightweight no-dependency fork from transformers.js (only tokenizers)

34

Hugging-Face-Supporter/tftokenizers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

10

HeegyuKim/any-tokenizers

Tokenizers for the any modalities

0

AILab-CVC/seed-tokenizer-2

Model

4

brian030128/transformers_

A fork from huggingface transformers.

0

SumanthRH/tokenization

A comprehensive deep dive into the world of tokens

231

huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

11,018

Daniel0724/OmniTokenizer

Model

1

Languages

JavaScript

100.0%

r4ghu/tokenizers.js

Port of the lightweight tokenizers.js from huggingface/transformers.js

0

stars

8

commits

JavaScript

primary language

Oct 20, 2024

updated

README

tokenizers.js

Port of the lightweight tokenizers.js from huggingface/transformers.js

Goal

  • Create a light weight tokenizers.js from @huggingface/transformers.js
  • Make transformers.js utilize this tokenizers.js
  • (If possible) transfer the project repo to Hugging Face. For example, @huggingface/tokenizers.js

Similar repos

lenML/tokenizers

a lightweight no-dependency fork from transformers.js (only tokenizers)

34

Hugging-Face-Supporter/tftokenizers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

10

HeegyuKim/any-tokenizers

Tokenizers for the any modalities

0

AILab-CVC/seed-tokenizer-2

Model

4

brian030128/transformers_

A fork from huggingface transformers.

0

SumanthRH/tokenization

A comprehensive deep dive into the world of tokens

231

huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

11,018

Daniel0724/OmniTokenizer

Model

1

Contributors

r4ghu

8 commits

Languages

JavaScript

100.0%