unremarkable

A tiny LLM running entirely on a reMarkable 2.

An inference engine written in C++23, built to learn how to make language models run faster on limited hardware. It starts as plain FP32 loops and gets faster one measured optimization at a time; each step is tagged (rung-01, rung-02, …) and explained in the optimization ladder. Runs SmolLM2-135M-Instruct, with optional tablet UI patches.

Ceiling

SmolLM2-135M, Q8, two cores: 2.708 GB/s ÷ 0.151 GB/token ≈ 17.9 tok/s memory-only ceiling.

• Calculation
• Benchmarks and changes

Batched prefill cuts the wait for the first token from 3.94 s to 2.64 s on our 26-token prompt.

Runs -- the turtle tracks decode speed.

Code and docs

• Inference: follow a token through memory and the model.
• Forward pass · Batched prefill · Kernels
• Optimization ladder: changes, measurements, and diffs.

Run locally

Requires a C++23 compiler, Make, Python 3.11+, and curl on Linux or macOS.

make
make models
./build/unremarkable models/stories15M.bin -z models/tokenizer.bin -i "Once upon a time"

For SmolLM2, run make chat-model-q8 (needs numpy), then use models/smollm2-135m-q8.bin with -z models/smollm2-135m-q8.tok -c 512 -j 2; it needs about 171 MiB. make chat-model writes the 538 MiB FP32 checkpoint the first three rungs were measured on; on the tablet it can exhaust memory while the UI runs.

make test-models
make check
make check-sanitize
make check-format

Build and deployment · Model files · Attribution