rodneylab/tokenator

Count the number of tokens in an LLM prompt

Rust

1,025 commits

updated Jul 23, 2026

See the code

llm

ollama

rust

tokens

README

tokenator

Count LLM prompt tokens to fine-tune prompts

Warning 🚧 Work in progress

License

The project is licensed under BSD 3-Clause License — see the LICENSE file for details.

Contributors

rodneylab

711 commits

dependabot[bot]

314 commits

Similar repos

ritenv/tokensift

Token-efficiency linter for LLM prompts and payloads

gsarti/labl

Token-level Label Management Toolkit

happytoolin/happytokenizer

Count your tokens and see how they're done

brexhq/prompt-engineering

Tips and tricks for working with Large Language Models like OpenAI's GPT-4.

9,586

ziyaow1010/Reasoning_Length_Prediction

microsoft/Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

213

daspartho/prompt-tokenizer

Model

Prompt Tokenizer

joshualu777/LLM-SpecDec

Languages

Rust

93.4%

Just

6.6%

rodneylab/tokenator

Count the number of tokens in an LLM prompt

Rust

1,025 commits

updated Jul 23, 2026

See the code

llm

ollama

rust

tokens

README

tokenator

Count LLM prompt tokens to fine-tune prompts

Warning 🚧 Work in progress

License

The project is licensed under BSD 3-Clause License — see the LICENSE file for details.

Similar repos

ritenv/tokensift

Token-efficiency linter for LLM prompts and payloads

gsarti/labl

Token-level Label Management Toolkit

happytoolin/happytokenizer

Count your tokens and see how they're done

brexhq/prompt-engineering

Tips and tricks for working with Large Language Models like OpenAI's GPT-4.

9,586

ziyaow1010/Reasoning_Length_Prediction

microsoft/Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

213

daspartho/prompt-tokenizer

Model

Prompt Tokenizer

joshualu777/LLM-SpecDec

Contributors

rodneylab

711 commits

dependabot[bot]

314 commits

Languages

Rust

93.4%

Just

6.6%