V100-LLM

One folder to run the best local LLMs on an NVIDIA Tesla V100 16 GB under Windows. It includes a GUI to pick a model and an engine ("kernel"), downloads anything that is missing, tunes every model for a 16 GB V100, gives each model a vision (image) mode, and ships a test suite that scores models in a leaderboard.

You don't need WSL, Docker, CUDA, Visual Studio or Python: the folder brings its own engines, CUDA runtime and Python.

Demo: MiniCPM-V 4.7 35B-A3B at ~90 tok/s on one V100 16 GB

Real-time screen recordings (not sped up) of the fast-128k preset: official OpenBMB mix_q3gud_im quant, stock llama.cpp, 128k context, vision enabled. The tokens/s counter under each reply is the chat app's own measurement.

The GUI (START-GUI.bat): pick a model, preset and engine, load or unload it, edit any parameter, chat with images, and watch live tok/s, VRAM, GPU load, CPU and RAM.

Leaderboard (hard suite v2)

72 tasks in 7 categories, every one graded by code (see Evaluating models). Scores are 0-100 per category, and Overall is the mean of the categories. Gen tok/s is the mean generation speed over replies of 64+ tokens; prompt tok/s is measured on short prompts. Every model runs with its recommended preset from catalog.json. Rebuild this table with EVALUATE.bat.

Notes:

• Long context is a ~14k-token text with three facts hidden in different places that have to be combined. Qwen3.8-27B PXQ3 runs with an 8k context preset, so the text does not fit; Ornith runs with 16k and still answered wrong.
• Fusion4 and Ornith run without the model card's repetition penalties. With them they scored much lower, because the penalties corrupt numbers, code and tool calls (8080 became 80). The penalties are still available as the creative preset.
• Fastest: MiniCPM-V 4.7 Q3 mix runs at 93 tok/s with 128k context on the 16 GB V100 (preset fast-128k): the vision encoder runs on the CPU, which frees enough VRAM to keep every expert on the GPU. A 120k-token prompt was filled without a crash (36 tok/s while generating at 120k).
• Strata runs three quantizations of the same 125B model (IQ2_XS, Q2_0 and the Coder IQ1_M with half the experts); they share one 28.8 GB n-gram table that is read from the SSD.
• MiniCPM-V CRACK with its DSpark draft model is slower on a 16 GB card (34 vs 38.5 tok/s): the drafter needs ~1.5 GB of VRAM, so 7 more layers of experts go to RAM, and chat drafts are accepted only ~47% of the time.
• ExLlamaV3 (Qwen3.8-27B EXL3) runs on the V100 but at ~11 tok/s, so it is not scored.
• The older, easier suite (v1) is still in eval/results/; most models scored 90-100 there, which is why v2 exists.

Quick start

1. Install an NVIDIA driver from the R576 series or newer. A V100 in TCC mode is fine.
2. Download this repository (green Code → Download ZIP, or git clone), then extract it, for example to D:\V100-LLM.
3. Double-click START-GUI.bat. The first run downloads the portable Python runtime (37 MB), then opens the GUI at http://127.0.0.1:7860.
4. Pick a model, then press Download missing. That fetches the model files from Hugging Face (checking each one's checksum) and the engine from this repo's release.
5. Press Start, then chat in the right-hand panel. A live tokens/s meter, prompt speed and image upload are built in. Any OpenAI-compatible app can also use http://127.0.0.1:8899/v1.

Every parameter is editable in the GUI. Recommended restores the tuned values, and Save .bat writes a standalone launcher for your own settings.

Models (tested on one Tesla V100-SXM2-16GB, Xeon E5-2660 v4, 64 GB DDR4)

Every preset is tuned to fill the 16 GB of VRAM (about 15-15.5 GB used). "System RAM used" is the MoE expert weights kept in RAM (--n-cpu-moe, measured from the GGUF tensors) plus ~1.5 GB for the engine itself; the vision encoder adds ~1 GB when it runs on the CPU. CPU threads: the MiniCPM-V presets use -t 14, the others the engine default (one per physical core). The CPU only does real work for models that keep experts or the vision encoder in RAM.

"Full-context test": a prompt of about 250,000 tokens with one secret code hidden in the middle. The model has to return it exactly. Models bigger than 16 GB (Nemotron, Strata, part of MiniCPM-V) keep some experts in system RAM; they need 32-64 GB of RAM, and their speed depends on the CPU and memory.

Some of these models are "uncensored" / "abliterated" fine-tunes with their safety tuning removed (Fusion4, HauhauCS, Bonsai Heretic). Their model cards say so.

Folder layout

START-GUI.bat            the GUI (model + kernel picker, downloads, chat, tokens/s)
EVALUATE.bat             score models and rebuild the leaderboard
catalog.json             every model: files (URL + SHA-256), tuned presets, compatible kernels, measured speeds
gui/                     launcher.py + ui.html (Python standard library only)
models/<model>/
    model/               the model files
    kernel/<engine>/     the engine for this model (hard links into _kernels/, so no extra disk space)
    run-<preset>.bat     standalone launchers (work without the GUI)
_kernels/<engine>/       shared engine store (filled by downloads)
runtime/python/          portable Python 3.12 (+ pylibs for Strata's server)
eval/                    test suite, results, leaderboard
tools/                   how the engines were built (patches, build scripts)

The folder is self-contained and can be copied to another PC or drive. Hard links fall back to copies when the target is not NTFS.

Engines (kernels)

CUDA 13 dropped Volta, so every engine here is built with CUDA 12.x for sm_70.

Evaluating models

EVALUATE.bat starts each installed model with its recommended preset (it starts the GUI launcher in the background if it is not running), runs the hard test suite (v2) and rebuilds eval/results_hard/leaderboard.html, .png and .md.

EVALUATE.bat                              every installed model that has no score yet
EVALUATE.bat saluki-27b                   one model
EVALUATE.bat saluki-27b:262k              one model, a specific preset
EVALUATE.bat saluki-27b --redo            score it again
EVALUATE.bat saluki-27b --redo --only=coding   re-run one category and merge it into the old result
EVALUATE.bat --basic                      the older, easier suite (v1, eval/results/)

Then runtime\python\python.exe tools\readme-leaderboard.py copies the new table into this README.

How the hard suite works

The first suite (v1) was too easy: most models scored 90-100, so it could not tell them apart. The hard suite (v2) is built so that no 16 GB-class model gets everything right. It has 72 tasks in 7 categories:

Rules that keep the results fair and repeatable:

• No LLM judge. Everything is checked by code, so the same answer always gets the same score.
• Deterministic and offline: temperature 0, thinking off, no internet access (web search uses the fixed index).
• Same preset as users get: each model runs with its recommended GUI preset, so the speed and score match what you see.
• Every task is proven solvable: eval/reference_solutions.py holds a correct solution for every coding and agent task, and selftest_hard.py / selftest_agent.py check that they pass the graders. Computed answers (dates, sums, codes) are asserted in tasks_hard.py.
• Answer extraction: the model may think out loud; the grader takes the final line. Tool calls are accepted in the OpenAI format and in the Qwen XML / <tool_call> text formats.
• Score: each category is the percentage of passed tasks; Overall is the mean of the 7 categories (each category weighs the same). Speed (gen tok/s) is the mean over replies of 64+ tokens during the run.

Every run is saved in eval/results_hard/<model>.json with each task's answer and pass/fail, so you can see why a model lost points.

Adding a new model:

1. Add an entry to tools/catalog-src.json: files (Hugging Face repo + file name) and a preset (kernel + parameters).
2. Run runtime\python\python.exe tools\make-catalog.py. It fills in sizes and SHA-256 sums.
3. Restart the GUI, download the model, press Start to check it works, then run EVALUATE.bat <id>.

Adding new tasks: add them to eval/tasks_hard.py (and a reference solution for coding/agent tasks), run selftest_hard.py, then re-score with EVALUATE.bat --redo --only=<category>.

Credits and licences

The engines are MIT (PXA, llama.cpp, Prism ML fork, Strata); see the LICENSES folder inside each kernel. The CUDA runtime DLLs are redistributed under the NVIDIA CUDA EULA, and the Microsoft Visual C++ runtime DLLs under the Visual Studio redistributable terms. Models are not part of this repository: they download from their authors' Hugging Face pages and fall under those licences. The scripts, GUI, patches and test suite in this repository are MIT licensed.