token-difr

Cheaply verify that LLM API providers are running the models they claim - $0.06 for Kimi K2.6!

The Problem

When you call an API claiming to serve "Llama 3.3 70B Instruct", how do you know that's actually what's running? Providers might have bugs, substitute smaller models, use aggressive quantization, or apply undisclosed modifications.

Traditional benchmarks are expensive and noisy. A single evaluation question might generate thousands of tokens but only counts as one data point. Evaluation results often vary by ±5-10% between runs with significant inference costs. See Why Benchmarking Is Hard for more on the challenges.

The Solution

With greedy decoding (temperature=0), identical models produce identical outputs (with some small divergence due to floating point noise). If a provider claims to run Model X, their token-by-token outputs should almost exactly match a trusted copy of Model X.

Token-level verification gets statistically significant results cheaply, because each token is an independent data point. 100 prompts × 200 output tokens is 20,000 samples, enough for match rates that are stable to a fraction of a percent between runs, for a few cents.

Two Workflows

The same measurement supports two jobs:

Snapshot verification generates outputs once from a trusted reference and saves the token IDs. To verify, just generate fresh tokens from the provider and compare against that snapshot. No GPU is needed for a check, and can be done on a laptop. Cost is often less than $0.01 per provider. However, it can only check short outputs of ~30 tokens.

Replay verification replays a provider's actual tokens through a reference model, scoring each position against what the reference would have produced given the provider's own preceding tokens. Because every position is judged independently, it can check an output of any length. However, it requires loading a model onto a GPU, which can be less convenient.

Installation

pip install token-difr

Requires Python >= 3.10 and an OpenRouter API key for generation. Replay verification additionally needs a reference backend — a Tinker API key, or a local GPU for vLLM.

export OPENROUTER_API_KEY="your-key"
export TINKER_API_KEY="your-key"

Quick Start: Snapshot Verification

import json
from token_difr import construct_prompts, generate_tinker_responses, prefix_match

MODEL = "openai/gpt-oss-20b"

# --- One-time: build a reference from a trusted source ---
prompts = construct_prompts(n_prompts=300, model_name=MODEL)
reference = generate_tinker_responses(prompts, model_name=MODEL, max_tokens=35)

with open("reference.json", "w") as f:
    json.dump(
        [{"conversation": p, "token_ids": s.output_token_ids}
         for p, s in zip(prompts, reference)],
        f,
    )

# --- Recurring: compare a provider against the stored tokens ---
with open("reference.json") as f:
    samples = json.load(f)

# provider_token_ids: outputs generated from the provider for the same prompts,
# re-encoded to token IDs. See demos/snapshot_verification_demo.py for the full flow.
result = prefix_match([s["token_ids"] for s in samples], provider_token_ids)

print(result)
# PrefixMatchResult(97.6% match rate, 6941 matched tokens across 300 sequences, 166 diverged)

A sweep of 300 prompts × 35 tokens costs roughly $0.003-$0.007 per provider for gpt-oss-20b, and match rates reproduce to within ±0.1 percentage points between runs.

See the snapshot verification guide for more details.

Quick Start: Replay Verification

from token_difr import audit_provider, construct_prompts

prompts = construct_prompts(
    n_prompts=100,
    model_name="meta-llama/Llama-3.3-70B-Instruct",
    system_prompt="You are a helpful assistant.",
)

result = audit_provider(
    prompts,
    model="meta-llama/Llama-3.3-70B-Instruct",
    provider="together",   # OpenRouter provider to test
    max_tokens=200,
)

print(result)
# AuditResult(98.3% match rate, 18421 tokens across 100 sequences)

AuditResult carries exact_match_rate (the primary metric), avg_prob, avg_margin, total_tokens, and n_sequences. When you need to localize a divergence rather than summarize it, the lower-level verify_outputs* functions return per-token records.

See the replay verification guide for more details.

Choosing Your Reference

Everything rests on trusting the reference, and there are two options on a convenience-versus-trust axis:

1. Local vLLM — you hold the weights, so there is nothing left to trust. Requires a CUDA GPU.
2. An inference API (Tinker, Fireworks, and similar): most convenient for development, but you are trusting one provider in order to check another. You can mitigate by checking the verification API against reference tokens, but you are then two hops away from ground truth.

See reference backend selection for the trade-offs and how to validate an API reference.

Interpreting Results

Match rates measure divergence from a reference, not quality. A high rate against a trusted reference is strong evidence the provider is running the claimed model, but several things cause legitimate divergence:

There is no universal threshold separating "good" from "bad" providers, and small gaps are not meaningful: any difference in the prompt, semantic or not, costs a point or two. Baseline first by running the same model across several providers, then treat a provider's drift against its own history as the signal worth acting on.

Model Registry

from token_difr import FIREWORKS_MODEL_REGISTRY, register_openrouter_model

# Only needed when the OpenRouter name differs from hf_name.lower()
register_openrouter_model(
    "Qwen/Qwen3-235B-A22B-Instruct-2507",
    "qwen/qwen3-235b-a22b-2507",
)

Limitations

• Temperature must be 0: Sampling seeds are not standardized across providers, so only greedy decoding produces comparable outputs.
• Tokenization edge cases: encode(decode(tokens)) may not equal the original tokens, causing ~0.5% of mismatches even for identical models.
• Prompt rendering is not under your control: providers apply chat templates server-side. Pin any settings the model exposes (gpt-oss reasoning effort, for example) on both sides, or a mismatch will look like a wrong model.
• Model availability: both OpenRouter and your chosen reference backend must serve the model.

Documentation

• Snapshot verification — recurring provider checks against stored reference snapshots
• Replay verification — per-token checking of individual outputs
• Adding a model or provider — pinning template settings and getting the response encoding right
• Reference backend selection — local model vs inference API, and how to validate one

Runnable examples live in demos/.

API Reference

Prompts and generation

• construct_prompts(n_prompts, model_name, ...) — load prompts from the WildChat dataset
• generate_tinker_responses(conversations, model_name, ...) — reference outputs as exact token IDs
• generate_openrouter_responses(client, conversations, model, provider, ...) — generate from a provider
• tokenize_openrouter_responses(conversations, responses, tokenizer, ...) — responses to TokenSequence
• encode_thinking_response(...) / encode_harmony_response(...) — rebuild a response's token IDs from text

Snapshot verification

• prefix_match(reference_token_ids, check_token_ids) — aggregate prefix comparison
• matched_prefix_length(a, b) — tokens matching before the first divergence

Replay verification

• audit_provider(conversations, model, provider, ...) — high-level generate-and-verify
• verify_outputs(sequences, model_name, ...) — local vLLM verification
• verify_outputs_fireworks(...) / verify_outputs_tinker(...) — API verification
• compute_metrics_summary(results) — aggregate per-token metrics

Model registry

• FIREWORKS_MODEL_REGISTRY, OPENROUTER_MODEL_REGISTRY
• register_fireworks_model(hf_name, fireworks_name), register_openrouter_model(hf_name, openrouter_name)
• get_openrouter_name(hf_name), guess_fireworks_name(hf_name)

Data classes

• TokenSequence(prompt_token_ids, output_token_ids)
• TokenMetrics(exact_match, prob, margin, logit_rank, gumbel_rank)
• AuditResult(exact_match_rate, avg_prob, avg_margin, ...)
• PrefixMatchResult(match_rate, n_sequences, n_diverged, ...)

License

MIT