Model Toolkit

Practical tools for evaluating, observing, pruning, and quantizing language models. Each tool also includes a portable SKILL.md for coding agents. Pick a folder, install that tool's dependencies, and run it. There is no server or orchestration framework to deploy.

This repository collects the workflows used in 0xSero's REAP and EXL3 experiments, including the GLM-5.3 fidelity campaign and Harbor coding evaluations. It contains reusable Python tools plus small launchers for the original upstream implementations. Model weights, benchmark questions, private session logs, and credentials are not bundled.

Choose a tool

“Intelligence” is measured through specific tasks, not a single invented composite score. DeepSWE is the concrete software benchmark included here; it is not labeled as a separate benchmark called “Deep Suite.”

Start here

Python 3.10+ is required. External tools may require newer Python; the pinned Cerebras REAP checkout requires Python 3.12+.

git clone https://github.com/0xSero/model-toolkit.git
cd model-toolkit
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python toolkit.py list

Most launchers and GPQA use only the Python standard library. NumPy is needed for KL, pruning plans, and route coverage. Install large GPU libraries and benchmark harnesses in separate environments using each folder's README; their dependencies may conflict.

First check a running model

export OPENAI_API_KEY='your-local-or-provider-key'
python toolkit.py run runtime-checks \
  --base-url http://localhost:8000/v1 --model your-served-model \
  --output runs/smoke.json

Then run a small evaluation

After accepting GPQA's dataset terms and downloading gpqa_diamond.csv:

python toolkit.py run gpqa-diamond \
  --csv data/gpqa_diamond.csv --model your-served-model \
  --base-url http://localhost:8000/v1 --limit 2 \
  --output runs/gpqa-smoke

Remove --limit for the full 198-question set. Configure sampling, reasoning effort, and output budget explicitly for your comparison. Request failures withhold the aggregate accuracy; output truncations are reported separately.

Preview an expensive command first

python toolkit.py run terminal-bench \
  --model openai/your-served-model --jobs-dir runs/tb21 \
  --limit 1 --dry-run
python toolkit.py run exl3-quantization \
  --input models/source --output models/exl3-4bpw \
  --work runs/exl3-work --bits 4 --dry-run

The previews do not require the external harness or a GPU. Actual runs do. Terminal-Bench and DeepSWE require a container environment. SciCode executes generated code: run its harness inside a disposable evaluation environment.

How the compression tools fit together

Pinned model + calibration corpus
             ↓
       REAP observations → coverage checks → expert keep plan
             ↓                                 ↓
       upstream pruning / architecture-specific weight surgery
             ↓
        EXL3 conversion → runtime checks → KL + task benchmarks
                                             ↓
                                      sealed evidence

Observation formats, expert tensor layouts, and quantization kernels vary by architecture. The tool READMEs state the supported boundary. A keep plan does not modify weights. A completed conversion does not prove runtime support or retained model quality.

For KL, compare the quantized model against pruned BF16 to isolate quantization loss, and separately against original BF16 to measure total pruning-plus-quantization loss. Keep sampling, prompts, tokenization, and task sets fixed when comparing task scores.

Existing Hugging Face work

• GLM-5.3 REAP fidelity study: criterion sweeps, keep plans, KL reports, and GPQA artifacts.
• REAP calibration data: calibration inputs and collection methodology. Audit constituent sources before claiming a benchmark-clean evaluation.
• Qwen layerwise observations and GLM layerwise observations: prior observation artifacts.
• GLM-5.3 observations: referenced by the fidelity study, but not anonymously accessible when checked on 2026-09-10. No tool here requires access to it.
• Cerebras REAP and PR #17: underlying observation and pruning work.

See REAP observation for the executable path and provenance for the connection to earlier sessions and skills.

Reproducibility and status

upstreams.json records exact upstream commits. python toolkit.py fetch NAME checks out one of them under .external/; it does not install dependencies. Record harness/package versions, model and dataset revisions, hardware, context, sampling, timeouts, and all failed attempts with your run.

The included tools are tested with synthetic data, mocked API responses, and CLI checks. This release does not claim a fresh GPU conversion, full benchmark, or architecture compatibility test. See validation. Historical model scores are deliberately not presented as results of this release.

Run the local checks:

python -m unittest discover -s tests -v

Credits and licensing

MIT for this toolkit's code; imported MIT notices are retained in licenses/. External harnesses, datasets, models, and model-specific code retain their own licenses. REAP belongs to Cerebras Research; ExLlamaV3/EXL3 to turboderp and contributors; Terminal-Bench/Harbor to their authors; DeepSWE/Pier to Datacurve; SciCode to its authors; lm-evaluation-harness to EleutherAI and contributors. See provenance.