hf-tokenizer-testing

A simple script to test if GPT tokenizers from various popular models can encode+decode documents losslessly.

Documents tested are:

• QQQ_0: Simple sanity test
• QQQ_1: A sample from TinyStories dataset that looks innocent, but most tokenizers seem to fail because of a little bit unusual punctuation.
• QQQ_2: Another, easier, sample from TinyStories
• QQQ_3: Markus Kuhn's UTF-8 stress test, with couple lines removed b/c python encoder couldn't handle them. Correctly encoding/decoding this document is a very good sign of "correctness".
• QQQ_4: Wikipedia page source for Snow Leopard
• QQQ_5: Book Pride and Prejudice from The Project Gutenberg
• QQQ_6: stb_image.h. C source code file
• QQQ_7: Japanese book from The Project Gutenberg, 苦悶の欄(Kumon no ran)

Why?

Does it matter if tokenizers can/can't reproduce the input exactly? I guess this is subjective, but I'd say it's at least a nice feature. A feature that some tokenizers out there don't seem to have.

NOTE: When decoding, it's important to pass clean_up_tokenization_spaces=False to tokenizer.decode. Otherwise you will get a lot of missmatches related to tokenizer eating whitespace.

Results

Pass Fail

Tokenized length

My Takeaways

• Llama tokenizer is really good. Small vocab, lossless, good compression, shown to work well in a real LLM. But downside with the Llama tokenizer is that it runs significantly slower than some of the other tokenizers (runtime speed is not shown here, but I've observed this elsewhere)
• Open Llama is ok in English but has drastically worse compression than Llama in the utf8 test and Japanese. In my opinion, this is a pretty bad look.
• Only other lossless tested here was bloom, but 250k vocab is kind of a lot for me. All the extra tokens lower the tokenized lengths, but is it worth it? Also there is some tradeoffs with regards to how the models learns rarely seen tokens.

Duplicates

Many models use the same tokenizers. Listing some of the relationships here for future reference to myself.

• gpt2: EleutherAI/gpt-j-6b, facebook/opt-66b (plus couple extra tokens), EleutherAI/gpt-neo-125m, roneneldan/TinyStories-8M
• EleutherAI/gpt-neox-20b: EleutherAI/pythia-2.8b,