Nemotron-3-Nano-4B — LiteRT-LM

nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Nemotron-3-Nano in LiteRT form.

A reasoning model (<think>, ChatML turns) on a three-kind hybrid stack: 21 Mamba2 selective-scan layers + 17 plain MLP layers + 4 grouped-query attention layers (42 total). Only the 4 attention layers keep KV (4096-token budget here), the mamba layers carry constant-size conv + SSM state, and the MLP layers carry no state at all — 50 state buffers in total (42 mamba, 8 KV), so memory stays nearly flat with context length.

Geometry: hidden 3136, 40 query / 8 KV heads, mamba 96 heads × 80 dim (state 128, conv 4, 8 groups), vocab 131,072, untied embeddings.

Correctness

All numbers below were measured on this exact file (litert-lm 0.16.0, Apple M4 Max).

• 8-question sanity gate: 7/8 on CPU, 8/8 on GPU, non-degenerate on both. The single CPU miss is the rhyme-completion item — it answers "violets are purple" where the gate wants "blue"; the GPU run answers "blue". Every arithmetic, factual, and translation item is correct on both backends.
• Chat template is byte-equal to the source: the embedded Jinja matches the repo's chat_template.jinja exactly (10,504 / 10,504 bytes). Note the source repo's tokenizer_config.json carries a different 10,497-byte copy; the bundle embeds the one AutoTokenizer actually resolves.
• Turn-end stop tokens are <|im_end|> (id 11) alongside the exported id 2.
• No spurious start token. The source tokenizer sets add_bos_token: False and the template never renders a leading BOS, so the <s> the bundler would otherwise prepend is dropped — the on-device token stream matches the training stream. Honest note: at this scale the model is robust either way (the gate scores 7/8 with the token and 7/8 without, and greedy decoding in PyTorch is byte-identical on 2 of 3 probes), so this is a correctness-of-convention fix rather than a rescue.

Usage

# CPU
litert-lm run ./Nemotron-3-Nano-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU — pass --cache no (see the honest note below)
litert-lm run ./Nemotron-3-Nano-4B_int8.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and the stock Nemotron-3-Nano chat template. Seven prefill signatures (1024/256/64/16/4/1 + decode) are exported so the runtime picks tight chunks.

Performance

litert-lm benchmark <file> -p 256 -d 256 --runs 3 --cache no, litert-lm 0.16.0, Apple M4 Max. Two independent runs:

Both figures per cell are the two runs, not a range estimate. GPU repeats within ~1.4%; CPU prefill spreads ~13%, partly because the host was not idle during these runs (another export was using the machine) — read the CPU column as an order of magnitude, not a precise figure. --cache no matters for more than tidiness here: with the compiled-graph cache the benchmark reports a much faster CPU prefill because it is not doing the same work.

Honest notes

• GPU requires --cache no on this bundle. With the compiled-graph cache enabled, litert-lm run --backend gpu fails with WebGPU Invalid BindGroup validation errors, and an 8-question sweep through the Mac verify harness returns token soup (0/8). The same file with --cache no answers 8/8. Measured as a one-variable comparison — same runner, same file, cache flag flipped — so the cache path is where it goes wrong; the root cause is not isolated further here.
• Not measured on a phone yet. The desktop numbers above are Mac-only. A 4B of this shape did not fit an 8 GB Android phone when the sibling Nemotron-H-4B was measured, so expect to need a higher-RAM device; that is an expectation carried over from a different bundle, not a measurement of this one.
• It is a reasoning model. Answers arrive after a <think> block, so give it a token budget that fits the thought (the gate above used 3200).
• int8 is applied to linears and the embedding only; the convolutions and the selective scan stay float, which is what keeps the hybrid state numerically sane.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch. One command, no per-model work — the reproduction script, the patch, and the full measurement record are in hf-to-litertlm:

python scripts/convert.py nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

Two things that route this model correctly and are worth knowing if you convert your own:

• auto_map in a config is not proof of remote code. This repo declares auto_map, but transformers registers nemotron_h natively, so without trust_remote_code the library implementation loads and the repo's Python is never imported. A converter that refuses on auto_map alone will refuse this model for no reason.
• ≥3B exports use a reduced 7-signature prefill ladder. Every exported signature costs engine RAM whether or not it is called, and a 4B hybrid with the full 11-signature ladder is exactly the shape that trips memory limits at GPU program init.

Conversion took 1645 s on an M4 Max. See REPRODUCE.md for the Nemotron-H family recipe and the measurements behind every claim on this card.

2026-08-31 — thought channel declared (metadata only, weights unchanged)

Nemotron-3-Nano-4B_int8.litertlm now declares the reasoning channel in its metadata (LlmMetadata.channels: channel name thought, markers <think>…</think> exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.