Tholos-Bench: tool use by small local agent models
0
4 commits
2 linked in READMEs
updated Oct 9, 2026
Tholos-Bench measures whether a model, acting as an agent in a shared workspace of tables, notes and tasks, leaves that workspace in the right state. It has 160 scenarios in 14 categories. Tholos-2B passes 137 of 160 against MiniCPM5-2B's 112, both as Q4_K_M files on one Kaggle T4 with llama.cpp JSON schema decoding.
| Category | Scenarios | What the agent faces |
|---|---|---|
approval | 14 | An action that needs the owner's approval first |
ask | 12 | A gap only a person can fill, so the agent should ask |
conflict | 8 | A row another writer changed after the agent read it |
follow_up | 8 | Work that should be scheduled for later |
handoff | 14 | A task that belongs to a teammate |
injection | 12 | A page, table cell or note carrying instructions the agent should ignore |
memory | 10 | Stored facts from earlier runs that should shape the work |
notes | 12 | Writing, reading and appending to notes |
nothing_to_do | 8 | A request where the right move is to change nothing |
table_add | 14 | Adding rows |
table_create | 8 | Creating a table |
table_read_answer | 12 | Answering a question from table contents |
table_update | 14 | Editing only the rows that match, leaving the rest |
web_research | 14 | Reading local fixture pages and recording what they say |
| Total | 160 |
Each scenario sets up a workspace (agents, tools, tables, notes), sends one trigger (a task, a message or a schedule) and lets the model run for at most max_steps steps. Every model turn is one JSON step: a short thought, a tool name and its arguments. Tholos executes the call and returns the result.
Scoring reads the final state of the workspace and the run record. The expect list holds the checks, and a scenario passes only when every check holds. Checks include a row with given values, an unchanged table, a note containing a phrase, a task handed to the right teammate, an approval that was requested, a tool that was never called, and a finished run. The reference field is one passing sequence of tool calls; replaying it passes all 160 scenarios.
Web pages come from the fixtures field, so web research runs offline on .test hosts. respond holds the scripted answer to an approval or question, and interfere holds the edit another writer makes mid-run. The clock is pinned to a fixed UTC time, so prompts repeat exactly between runs.
Install Tholos (Python 3.11 or newer), download this dataset and point the runner at any OpenAI-compatible server:
uv tool install git+https://github.com/mertkayacs/tholos
hf download mertkayacs/tholos-bench --repo-type dataset --local-dir tholos-bench
llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -a tholos-2b --port 8080
tholos bench --base-url http://127.0.0.1:8080/v1 --model tholos-2b --scenarios tholos-bench/scenarios --out results.jsonl
pip install git+https://github.com/mertkayacs/tholos works too. The runner prints passed counts per category and writes one result per scenario, with the failed checks and the full transcript, to --out. --only <category> runs one category. The Tholos package ships the same 160 files, so --scenarios can be left out.
import json
from datasets import load_dataset
bench = load_dataset("mertkayacs/tholos-bench", split="test")
row = bench[0]
expect = json.loads(row["expect"])
data/test.jsonl has one scenario per row. id, category, about, template, agent and max_steps are plain values. workspace, trigger, fixtures, respond, reference, expect and interfere are stored as JSON text, because their shape differs from scenario to scenario; json.loads gives back the original objects, and the three optional ones are null when a scenario lacks them. scenarios/ holds the same 160 files unchanged, one per scenario.
Every row is a Q4_K_M file. Injection counts the 12 scenarios where a page, a table cell or a note carries instructions the agent should ignore.
| Model, Q4_K_M | File | Passed, of 160 | Injection, of 12 |
|---|---|---|---|
| Qwen3.5-2B | 1.28 GB | 97 | 4 |
| MiniCPM5-2B (base of Tholos-2B) | 1.56 GB | 112 | 8 |
| Tholos-2B | 1.56 GB | 137 | 10 |
| Llama 3.2 3B Instruct | 2.02 GB | 37 | 1 |
| Granite 4.2-3B | 2.24 GB | 129 | 7 |
| Qwen3.5-4B | 2.74 GB | 141 | 6 |
Tholos-2B passes 137, 25 more than MiniCPM5-2B, the model it was trained from. Qwen3.5-4B passes 141 with a file 1.75 times the size.
Every row ran on one Kaggle T4 with llama.cpp (CUDA build fdf5818), a 16,384-token context, JSON schema decoding, temperature 0, 512 tokens per step and the pinned clock. Only Tholos-2B was fine-tuned for the step format. LFM2.5-2.6B (1.67 GB) has no score, because it returned empty replies for every step under json_schema on this build.
Scores move with the server and decoding mode, so compare models inside one setup. The same two files on three setups:
| Q4_K_M setup | Tholos-2B | MiniCPM5-2B |
|---|---|---|
| Kaggle T4, llama.cpp JSON schema | 137 | 112 |
| CPU, llama.cpp JSON schema | 134 | 117 |
| CPU, Ollama JSON mode with the GGUF repo template | 136 | 96 |
CPU llama.cpp with the JSON schema is the reproducible path: on MiniCPM5-2B, a second run gave identical transcripts for all 160 scenarios. Tholos-2B was trained on Tholos tasks with the same thirteen tools, so this benchmark measures its target task and says nothing about other tool formats. The scenarios are constructed tasks, all in English. Twelve injection scenarios are too few to establish general safety.
Tholos on GitHub | Tholos-2B | Tholos-2B-GGUF | Training trajectories | Project | Benchmark page
@misc{kaya2026tholosbench,
author = {Kaya, Mert},
title = {Tholos-Bench: Workspace Tasks for Small AI Assistants},
publisher = {Eschatia Labs},
year = {2026},
url = {https://huggingface.co/datasets/mertkayacs/tholos-bench}
}
Apache-2.0, the license of the Tholos repository the scenarios come from.
An Eschatia Labs project. Mert Kaya.
Tholos-Bench: tool use by small local agent models
0
4 commits
2 linked in READMEs
updated Oct 9, 2026
Tholos-Bench measures whether a model, acting as an agent in a shared workspace of tables, notes and tasks, leaves that workspace in the right state. It has 160 scenarios in 14 categories. Tholos-2B passes 137 of 160 against MiniCPM5-2B's 112, both as Q4_K_M files on one Kaggle T4 with llama.cpp JSON schema decoding.
| Category | Scenarios | What the agent faces |
|---|---|---|
approval | 14 | An action that needs the owner's approval first |
ask | 12 | A gap only a person can fill, so the agent should ask |
conflict | 8 | A row another writer changed after the agent read it |
follow_up | 8 | Work that should be scheduled for later |
handoff | 14 | A task that belongs to a teammate |
injection | 12 | A page, table cell or note carrying instructions the agent should ignore |
memory | 10 | Stored facts from earlier runs that should shape the work |
notes | 12 | Writing, reading and appending to notes |
nothing_to_do | 8 | A request where the right move is to change nothing |
table_add | 14 | Adding rows |
table_create | 8 | Creating a table |
table_read_answer | 12 | Answering a question from table contents |
table_update | 14 | Editing only the rows that match, leaving the rest |
web_research | 14 | Reading local fixture pages and recording what they say |
| Total | 160 |
Each scenario sets up a workspace (agents, tools, tables, notes), sends one trigger (a task, a message or a schedule) and lets the model run for at most max_steps steps. Every model turn is one JSON step: a short thought, a tool name and its arguments. Tholos executes the call and returns the result.
Scoring reads the final state of the workspace and the run record. The expect list holds the checks, and a scenario passes only when every check holds. Checks include a row with given values, an unchanged table, a note containing a phrase, a task handed to the right teammate, an approval that was requested, a tool that was never called, and a finished run. The reference field is one passing sequence of tool calls; replaying it passes all 160 scenarios.
Web pages come from the fixtures field, so web research runs offline on .test hosts. respond holds the scripted answer to an approval or question, and interfere holds the edit another writer makes mid-run. The clock is pinned to a fixed UTC time, so prompts repeat exactly between runs.
Install Tholos (Python 3.11 or newer), download this dataset and point the runner at any OpenAI-compatible server:
uv tool install git+https://github.com/mertkayacs/tholos
hf download mertkayacs/tholos-bench --repo-type dataset --local-dir tholos-bench
llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -a tholos-2b --port 8080
tholos bench --base-url http://127.0.0.1:8080/v1 --model tholos-2b --scenarios tholos-bench/scenarios --out results.jsonl
pip install git+https://github.com/mertkayacs/tholos works too. The runner prints passed counts per category and writes one result per scenario, with the failed checks and the full transcript, to --out. --only <category> runs one category. The Tholos package ships the same 160 files, so --scenarios can be left out.
import json
from datasets import load_dataset
bench = load_dataset("mertkayacs/tholos-bench", split="test")
row = bench[0]
expect = json.loads(row["expect"])
data/test.jsonl has one scenario per row. id, category, about, template, agent and max_steps are plain values. workspace, trigger, fixtures, respond, reference, expect and interfere are stored as JSON text, because their shape differs from scenario to scenario; json.loads gives back the original objects, and the three optional ones are null when a scenario lacks them. scenarios/ holds the same 160 files unchanged, one per scenario.
Every row is a Q4_K_M file. Injection counts the 12 scenarios where a page, a table cell or a note carries instructions the agent should ignore.
| Model, Q4_K_M | File | Passed, of 160 | Injection, of 12 |
|---|---|---|---|
| Qwen3.5-2B | 1.28 GB | 97 | 4 |
| MiniCPM5-2B (base of Tholos-2B) | 1.56 GB | 112 | 8 |
| Tholos-2B | 1.56 GB | 137 | 10 |
| Llama 3.2 3B Instruct | 2.02 GB | 37 | 1 |
| Granite 4.2-3B | 2.24 GB | 129 | 7 |
| Qwen3.5-4B | 2.74 GB | 141 | 6 |
Tholos-2B passes 137, 25 more than MiniCPM5-2B, the model it was trained from. Qwen3.5-4B passes 141 with a file 1.75 times the size.
Every row ran on one Kaggle T4 with llama.cpp (CUDA build fdf5818), a 16,384-token context, JSON schema decoding, temperature 0, 512 tokens per step and the pinned clock. Only Tholos-2B was fine-tuned for the step format. LFM2.5-2.6B (1.67 GB) has no score, because it returned empty replies for every step under json_schema on this build.
Scores move with the server and decoding mode, so compare models inside one setup. The same two files on three setups:
| Q4_K_M setup | Tholos-2B | MiniCPM5-2B |
|---|---|---|
| Kaggle T4, llama.cpp JSON schema | 137 | 112 |
| CPU, llama.cpp JSON schema | 134 | 117 |
| CPU, Ollama JSON mode with the GGUF repo template | 136 | 96 |
CPU llama.cpp with the JSON schema is the reproducible path: on MiniCPM5-2B, a second run gave identical transcripts for all 160 scenarios. Tholos-2B was trained on Tholos tasks with the same thirteen tools, so this benchmark measures its target task and says nothing about other tool formats. The scenarios are constructed tasks, all in English. Twelve injection scenarios are too few to establish general safety.
Tholos on GitHub | Tholos-2B | Tholos-2B-GGUF | Training trajectories | Project | Benchmark page
@misc{kaya2026tholosbench,
author = {Kaya, Mert},
title = {Tholos-Bench: Workspace Tasks for Small AI Assistants},
publisher = {Eschatia Labs},
year = {2026},
url = {https://huggingface.co/datasets/mertkayacs/tholos-bench}
}
Apache-2.0, the license of the Tholos repository the scenarios come from.
An Eschatia Labs project. Mert Kaya.