Baseten Tokenizer

High-performance, Rust-backed BPE tokenization for inference. The Python package is named basetenkenizer.

Install

pip install basetenkenizer

Loading by Hugging Face model ID downloads tokenizer.json on first use and then uses the local Hugging Face cache. Private or gated models use the usual HF_TOKEN environment variable.

Encode text

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
encoding = tokenizer.encode(
    "A very long prompt that is now much faster.",
    add_special_tokens=False,
)

print(encoding.ids)
print(tokenizer.decode(encoding.ids))

Tokenizer.from_file("tokenizer.json") loads a local tokenizer. Encoding objects expose ids, attention_mask, type_ids, and special_tokens_mask; selected fields can be moved into NumPy arrays with encoding.into_numpy(...).

Kimi K2.7 Code

Load the published tokenizer directly from its model repository:

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("moonshotai/Kimi-K2.7-Code")
ids = tokenizer.encode("def hello():\n    return 'world'").ids

For a chat prompt, preserve the boundary between template control tokens and untrusted message content with encode_segments:

user_message = "Write a Python HTTP server."

segments = [
    ("<|im_user|>user<|im_middle|>", True),
    (user_message, False),
    ("<|im_end|>", True),
    ("<|im_assistant|>assistant<|im_middle|><think>", True),
]

encoding = tokenizer.encode_segments(segments)
input_ids = encoding.ids

This is the minimal Kimi K2.7 Code user/assistant shape. Applications using system messages, tools, images, or existing assistant messages should render the complete official model template and retain the same control-text versus message-text boundaries.

Kimi K3

Kimi K3 uses XTML and a Python chat renderer rather than a Jinja chat template. Raw text can still be encoded normally:

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("moonshotai/Kimi-K3")
ids = tokenizer.encode("Hello from Kimi K3").ids

For rendered chat, pass the official Kimi K3 renderer's EncodeSegment objects to encode_segments as (segment.text, segment.allow_special). A minimal user message followed by an assistant-generation prefix looks like this:

user_message = "Explain speculative decoding."

segments = [
    ("<|open|>", True),
    ("message", False),
    (" role", False),
    ('="', False),
    ("user", False),
    ('"', False),
    ("<|sep|>", True),
    (user_message, False),
    ("<|close|>", True),
    ("message", False),
    ("<|sep|>", True),
    ("<|end_of_msg|>", True),
    ("<|open|>", True),
    ("message", False),
    (" role", False),
    ('="', False),
    ("assistant", False),
    ('"', False),
    ("<|sep|>", True),
    ("<|open|>", True),
    ("response", False),
    ("<|sep|>", True),
]

encoding = tokenizer.encode_segments(segments)
input_ids = encoding.ids

Use Kimi K3's official build_chat_segments renderer for production conversations involving tools, images, reasoning content, response schemas, or multiple message types. Do not concatenate its segments before encoding: doing so loses the special-token safety boundary.

Why encode_segments?

Each segment is a (text, allow_special) pair:

• allow_special=True recognizes tokenizer control tokens emitted by a trusted chat renderer.
• allow_special=False treats control-token-looking strings in user, tool, or attribute content as ordinary text.

tiktoken_safe=True is the default. It reproduces the chunk boundaries used by legacy tiktoken tokenizers, including on very long inputs, so token IDs stay compatible. Set it to False only when exact tiktoken parity is not required and whole-segment BPE encoding is intentional.

Post-processing, truncation, and padding are applied after all segment IDs have been joined.

Use with Transformers

Call patch_transformers before loading a tokenizer:

import basetenkenizer

basetenkenizer.patch_transformers()

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-120b")
tokens = tokenizer("Hello, world!")

Transformers v4 and v5 are supported. Pass patch_transformers(apply_chat_template=True) to also use the native renderer for supported render-only apply_chat_template(..., tokenize=False) calls; unsupported templates automatically fall back to Transformers.

Baseten Tokenizer is focused on inference and does not implement every Hugging Face Tokenizers training or alignment feature.

License

The basetenkenizer package is licensed under the Apache License 2.0. Third-party and derived components remain subject to their respective terms; see THIRD_PARTY_LICENSES.md for the release audit. Some files are vendored from other projects for testing, including files under vendored_tokenizers/, and remain subject to their respective terms rather than Apache-2.0. Some test and benchmark files contain or were bundled from internal implementations; they carry separate terms in their file headers and are not available under Apache-2.0. Non-Apache materials are excluded from published source and binary packages.

View basetenkenizer on PyPI

Acknowledgements and upstream contributions

Basetenkenizer builds on work from Crusoe's fastokens, gigatoken, and MiniJinja. We're proud to contribute to the open-source community and grateful for the work that made this project possible. In addition to these libraries, various other libraries are used as optional or required dependencies.

While developing Basetenkenizer, we also contributed improvements upstream:

• crusoecloud/fastokens#8: add CI for cross-compiled PyO3 builds and upgrade PyO3
• crusoecloud/fastokens#9: add a Tokio and PyO3 async interface
• mitsuhiko/minijinja#913: render None and booleans like Jinja2
• mitsuhiko/minijinja#912: reset loop-local variables between iterations
• huggingface/tokenizers#1968: add a PCRE2 backend and regex sharding for faster tokenization