dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScript

1,701

59 commits

updated Apr 24, 2025

tiktokenizer.vercel.app

See the code

README

Tiktokenizer

Online playground for openai/tiktoken, calculating the correct number of tokens for a given prompt.

Special thanks to Diagram for sponsorship and guidance.

https://user-images.githubusercontent.com/1443449/222598119-0a5a536e-6785-44ad-ba28-e26e04f15163.mp4

Acknowledgments

chatgpt

nextjs

openai

t3-stack

tiktoken

tokenizer

Significant stargazers

Peter Dave Hello

1,871 followers · starred Apr 2023

Omar Sanseviero

1,108 followers · starred Jul 2024

gaurav

289 followers · starred Feb 2025

Matt Pocock

49,598 followers · starred Feb 2025

Similar repos

dqbd/tiktoken

JS port and JS/WASM bindings for openai/tiktoken

1,074

same org

happytoolin/happytokenizer

Count your tokens and see how they're done

lucaspanizio/ai-tokenizer

Demonstração interativa de como diferentes modelos de IA tokenizam texto.

rodneylab/tokenator

Count the number of tokens in an LLM prompt

johannschopplich/tokenx

📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle

181

niieani/gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*,…

854

pkoukk/tiktoken-go

go version of tiktoken

955

Likheet/tokenizer-lab