Approximate LLM inference: replace exact weight-matrix multiplication with importance-sampled Monte Carlo estimates (exact top contributions + sampled tail), trading a controllable amount of accuracy for fewer weight bytes read.
Everything needed to reproduce our numbers is in this repository: pinned source, pinned model files (by sha256), pinned dataset, build and run scripts, and the raw results of every run we report. See REPRODUCE.md, and docs/REPRODUCE_70B.md to time Llama-3.3-70B against vanilla Ollama on one GPU.
Every speedup was measured on the machines and software versions named in the paper. Rerun on a similar machine with the same versions for a consistent check; we do not guarantee the same performance on other machines, versions or settings.
scripts/reproduce/retest.sh).After the paper is published, the engine will be released as Sushila.cpp (Scalable Upstream Synthesis for Hybrid Inference in Large-model Acceleration): llama.cpp and an Ollama build with precomputed landscapes, draft heads and kernels, using the same model files, commands and API.
Linux, macOS and Windows (WSL2): see INSTALL.md.
git clone https://github.com/syncaissa/sushila.cpp.git
cd sushila.cpp
scripts/setup.sh # build tools (Ubuntu 22.04)
scripts/build.sh # llama.cpp, CUDA if available
scripts/get_model.sh llama3.1-8b-q4km # from the Ollama registry, sha256-verified
scripts/get_data.sh # WikiText-2, sha256-verified
scripts/run_baseline.sh llama3.1-8b-q4km
Runs on any Linux machine or cloud VM, on Google Colab, or on RunPod; REPRODUCE.md covers each, and how to split a sweep over several machines.
| Part | Language | What it is |
|---|---|---|
Sushila Engine (sushila, sushila.exe; cli/) | Rust (tokio, axum, reqwest) | The one program people run: command line, local web server and OpenAI-compatible API, queue, installer of the engines and model packs (signed, SHA-256 checked), self-update, internet link, license check. It does not run the models itself: it starts the inference engines below as separate processes and talks to them over local HTTP. |
| Inference engines (what the name Sushila.cpp refers to) | C / C++ (+ CUDA, Vulkan, Metal) | llama.cpp for chat and code, with our landscape kernels (llama.cpp/); stable-diffusion.cpp for pictures and video, with precomputed cache plans; acestep.cpp for music, with our patches (hoststation/patches/acestep/). Prebuilt per operating system and GPU and installed by the engine. |
NVIDIA picture runtime (hoststation/runtime/image-nunchaku/) | Python | diffusers with Nunchaku's SVDQuant 4-bit kernels for Z-Image on NVIDIA GPUs; the engine runs it as one more local server. |
Sushila Station (station/) | Rust (Tauri 2) + HTML / CSS / JavaScript | The desktop app. Rust: window, tray, updates, starting the embedded engine; JavaScript: the screens (station/dist/), which the engine also serves as its page (one source of screens). |
sushila.ai (website/worker.js) | JavaScript (Cloudflare Worker) | Website, downloads, sign-in, sharing, model-pack catalog, releases, license-check records (DynamoDB, B2). |
Research and release pipeline (scripts/) | Python (+ shell) | Landscapes, draft heads, cache plans, benchmarks, every table in the paper. |
| Path | Contents |
|---|---|
llama.cpp/ | llama.cpp source, tag b11232 (6f767fe960c3b97cf37fac4626c86400561ca1e4), the version Ollama pins in ollama/LLAMA_CPP_VERSION. Upstream: https://github.com/ggml-org/llama.cpp (MIT) |
ollama/ | Ollama source at 1abe35e6e6e777e858bbfbba283667ee8d516801. Upstream: https://github.com/ollama/ollama (MIT) |
cli/ | the Sushila Engine (sushila, Rust) and the page it serves (cli/web/) |
station/ | Sushila Station, the desktop app (Rust, Tauri 2; screens in station/dist/) |
hoststation/ | engine runtimes and patches: the NVIDIA picture server (Python), acestep.cpp patches |
scripts/ | Setup, build, download and benchmark scripts |
configs/models.tsv | Models used, with Ollama registry reference, sha256 and license |
runpod/ | Create, list and stop RunPod CPU and GPU pods from the command line (optional) |
notebooks/ | colab_sweep.ipynb: the same experiments on Google Colab |
results/ | One folder per run: environment, raw logs, summary table |
website/ | worker.js: the sushila.ai site as one Cloudflare Worker |
assets/logo/ | the Sushila logo: SushilaLogoWithBaseG.jpg (original), its animation SushilaLogoWithBaseG.mp4, transparent icons (512, 256, 180, 64, 32 px), swan and base layers for the rocking logo; old/ = earlier versions |
docs/ | REPRODUCE_70B.md: Llama-3.3-70B against vanilla Ollama; BENCHMARK_PIPELINE.md: every program of that benchmark, step by step; ADD_A_MODEL.md: the same for any model, one config file each |
scripts/precompute/ | the per-model pipeline: precomputed draft head plus the comparison against vanilla Ollama (models/*.env, run_model.sh) |
The first commit in this repository is the unmodified upstream source of both
projects. All Monte Carlo changes come in later commits, so
git diff fbcddad -- llama.cpp ollama shows exactly what was changed.
Ollama builds llama.cpp from source and applies
ollama/llama/compat/001-llama-cpp-hooks.patch. Point it at this repo's copy
instead of fetching upstream:
OLLAMA_LLAMA_CPP_SOURCE=$PWD/llama.cpp cmake -S ollama/llama/server --preset cpu
mc-matmul.c): exact-mode output matches legacy
bit-for-bit; ablation modes zeros, placebo, topkscripts/sweep_mc.sh)mc_fraction option in Ollama (0 = stock Ollama, verified token-for-token)llama.cpp/ and ollama/ keep their upstream MIT licenses (see their LICENSE files).
Approximate LLM inference: replace exact weight-matrix multiplication with importance-sampled Monte Carlo estimates (exact top contributions + sampled tail), trading a controllable amount of accuracy for fewer weight bytes read.
Everything needed to reproduce our numbers is in this repository: pinned source, pinned model files (by sha256), pinned dataset, build and run scripts, and the raw results of every run we report. See REPRODUCE.md, and docs/REPRODUCE_70B.md to time Llama-3.3-70B against vanilla Ollama on one GPU.
Every speedup was measured on the machines and software versions named in the paper. Rerun on a similar machine with the same versions for a consistent check; we do not guarantee the same performance on other machines, versions or settings.
scripts/reproduce/retest.sh).After the paper is published, the engine will be released as Sushila.cpp (Scalable Upstream Synthesis for Hybrid Inference in Large-model Acceleration): llama.cpp and an Ollama build with precomputed landscapes, draft heads and kernels, using the same model files, commands and API.
Linux, macOS and Windows (WSL2): see INSTALL.md.
git clone https://github.com/syncaissa/sushila.cpp.git
cd sushila.cpp
scripts/setup.sh # build tools (Ubuntu 22.04)
scripts/build.sh # llama.cpp, CUDA if available
scripts/get_model.sh llama3.1-8b-q4km # from the Ollama registry, sha256-verified
scripts/get_data.sh # WikiText-2, sha256-verified
scripts/run_baseline.sh llama3.1-8b-q4km
Runs on any Linux machine or cloud VM, on Google Colab, or on RunPod; REPRODUCE.md covers each, and how to split a sweep over several machines.
| Part | Language | What it is |
|---|---|---|
Sushila Engine (sushila, sushila.exe; cli/) | Rust (tokio, axum, reqwest) | The one program people run: command line, local web server and OpenAI-compatible API, queue, installer of the engines and model packs (signed, SHA-256 checked), self-update, internet link, license check. It does not run the models itself: it starts the inference engines below as separate processes and talks to them over local HTTP. |
| Inference engines (what the name Sushila.cpp refers to) | C / C++ (+ CUDA, Vulkan, Metal) | llama.cpp for chat and code, with our landscape kernels (llama.cpp/); stable-diffusion.cpp for pictures and video, with precomputed cache plans; acestep.cpp for music, with our patches (hoststation/patches/acestep/). Prebuilt per operating system and GPU and installed by the engine. |
NVIDIA picture runtime (hoststation/runtime/image-nunchaku/) | Python | diffusers with Nunchaku's SVDQuant 4-bit kernels for Z-Image on NVIDIA GPUs; the engine runs it as one more local server. |
Sushila Station (station/) | Rust (Tauri 2) + HTML / CSS / JavaScript | The desktop app. Rust: window, tray, updates, starting the embedded engine; JavaScript: the screens (station/dist/), which the engine also serves as its page (one source of screens). |
sushila.ai (website/worker.js) | JavaScript (Cloudflare Worker) | Website, downloads, sign-in, sharing, model-pack catalog, releases, license-check records (DynamoDB, B2). |
Research and release pipeline (scripts/) | Python (+ shell) | Landscapes, draft heads, cache plans, benchmarks, every table in the paper. |
| Path | Contents |
|---|---|
llama.cpp/ | llama.cpp source, tag b11232 (6f767fe960c3b97cf37fac4626c86400561ca1e4), the version Ollama pins in ollama/LLAMA_CPP_VERSION. Upstream: https://github.com/ggml-org/llama.cpp (MIT) |
ollama/ | Ollama source at 1abe35e6e6e777e858bbfbba283667ee8d516801. Upstream: https://github.com/ollama/ollama (MIT) |
cli/ | the Sushila Engine (sushila, Rust) and the page it serves (cli/web/) |
station/ | Sushila Station, the desktop app (Rust, Tauri 2; screens in station/dist/) |
hoststation/ | engine runtimes and patches: the NVIDIA picture server (Python), acestep.cpp patches |
scripts/ | Setup, build, download and benchmark scripts |
configs/models.tsv | Models used, with Ollama registry reference, sha256 and license |
runpod/ | Create, list and stop RunPod CPU and GPU pods from the command line (optional) |
notebooks/ | colab_sweep.ipynb: the same experiments on Google Colab |
results/ | One folder per run: environment, raw logs, summary table |
website/ | worker.js: the sushila.ai site as one Cloudflare Worker |
assets/logo/ | the Sushila logo: SushilaLogoWithBaseG.jpg (original), its animation SushilaLogoWithBaseG.mp4, transparent icons (512, 256, 180, 64, 32 px), swan and base layers for the rocking logo; old/ = earlier versions |
docs/ | REPRODUCE_70B.md: Llama-3.3-70B against vanilla Ollama; BENCHMARK_PIPELINE.md: every program of that benchmark, step by step; ADD_A_MODEL.md: the same for any model, one config file each |
scripts/precompute/ | the per-model pipeline: precomputed draft head plus the comparison against vanilla Ollama (models/*.env, run_model.sh) |
The first commit in this repository is the unmodified upstream source of both
projects. All Monte Carlo changes come in later commits, so
git diff fbcddad -- llama.cpp ollama shows exactly what was changed.
Ollama builds llama.cpp from source and applies
ollama/llama/compat/001-llama-cpp-hooks.patch. Point it at this repo's copy
instead of fetching upstream:
OLLAMA_LLAMA_CPP_SOURCE=$PWD/llama.cpp cmake -S ollama/llama/server --preset cpu
mc-matmul.c): exact-mode output matches legacy
bit-for-bit; ablation modes zeros, placebo, topkscripts/sweep_mc.sh)mc_fraction option in Ollama (0 = stock Ollama, verified token-for-token)llama.cpp/ and ollama/ keep their upstream MIT licenses (see their LICENSE files).