JevGPT

A classifier pretending to be a chatbot.

A deliberately questionable chatbot: Jev chooses one OpenAI tokenizer fragment at a time, and the growing answer becomes the next classification input. No generative model proposes the output.

Demo

Watch the MP4 · Record it yourself

A real two-turn chat with live Jev responses, played at original speed.

Methodology

Jev is the core: it selects every candidate and decides which token to append. Speculative drafting and the tokenizer support that process.

Architecture

Speculative mode: draft in parallel → verify → append. Four disjoint, stratified samples cover 1,016 tokens. Each Jev draft proposes one candidate for the same next-token position; the verifier selects one of the four candidates or DONE.

The main motivation for speculative mode is to expand the candidate vocabulary at each step: from a single 254-token shortlist to 1,016 distinct candidates across four samples. Each parallel draft predicts a possible next token from its sample, and the verifier selects which prediction advances the reply. The four predictions are alternatives for the same next position, rather than a sequence of four consecutive tokens.

Each shortlist mixes tokens from the prompt and recent reply, common corpus tokens, n-gram continuations, lexical matches, and random exploration. Candidates are distributed across the four samples without duplicates; unused slots are filled with random tokens from the remaining vocabulary. Each sample is shuffled and rebuilt at every output step.

See the experiment log for the iterations, observed outputs, latency measurements, and why we changed direction.

Run

Requires Python 3.11+ and uv.

uv sync
# Only if you do not already have a .env file:
cp -n .env.template .env
# Add your TypeSafe key to .env, then:
uv run jevgpt "Say hello in one short sentence."
uv run jevgpt

Interactive mode supports /reset and /quit. Output streams as tokens arrive. By default, the CLI shows only the conversation. Add --trace to show routing diagnostics and a final summary of stopping reason, generated tokens, HTTP calls, reported input-token usage, and elapsed time. The key stays server-side in this local process; .env is ignored by Git.

uv run jevgpt "Why is the sky blue?" --max-tokens 64
uv run jevgpt "Why is the sky blue?" --selection shortlist
uv run jevgpt "Hello" --dry-run             # No Jev calls or API key needed
uv run jevgpt --corpus ./my-conversations.txt --seed 42
uv run pytest

The first run downloads the cl100k_base tokenizer data. --dry-run may therefore need network access on its first use. No OpenAI API key is needed.

Selection modes

All modes use the prompt and full reply so far. Speculative mode makes four concurrent draft requests, then one verifier request. --selection speculative and --selection parallel select this same algorithm. Use --selection shortlist for the original one-call-per-token baseline. --selection hierarchical and --selection tournament remain available for experiments; tournament still launches all batches concurrently by default. Normal output stays a plain chatbot. --trace enables diagnostics. Retries and payload splitting can add calls.

Related reading: Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman, and Matias, 2023). Our Speculative mode uses four competing candidates for one position; it does not implement the paper’s distribution-preserving speculative decoding algorithm.

Tradeoffs

• Next-token predictions: all four drafts predict a continuation at the same position; the verifier selects one to append. Both draft and verifier use Jev. This candidate-selection procedure differs from the paper's sequential drafting and acceptance algorithm.
• Coverage vs. speed: tournament evaluates every eligible token at every step and is expensive. Hierarchy prunes branches, and shortlist is fastest but restricts candidates.
• Search isn't certainty: all approaches can discard a good continuation. Reported probabilities apply only to the choices shown, not the whole vocabulary.
• Text has limits: partial UTF-8 tokens are excluded, reducing multilingual coverage. Long conversations drop older turns; the current reply is retained.
• Quality is experimental: wider search hasn't demonstrated reliable improvement. One sky-blue answer was Because the Air air air blocks blue—still wrong and repetitive.

Generation stops on DONE, the output cap (128 tokens by default), or repeated loops. Local request-size guards keep payloads bounded but aren't exact Jev token counts.

Speculative mode and benchmark · Tournament details and benchmark · Other algorithms and live comparisons · TypeSafe API · Model limits