🦞 ClawEval — Can Your Local or Cloud LLM Actually Do the Job?

The only deterministic benchmark that tests LLMs as real AI agents — not trivia, not chat, not vibes. 59 specialized roles. Hard, verifiable tasks. Every score is reproducible. Built for OpenClaw, Hermes Agent, and any model-agnostic agent framework.

🔥 New models added regularly. Star ⭐ this repo to get notified when new results drop.

📬 Subscribe to AIgenteur Dispatch — benchmark results, local AI guides, and agent workflows in your inbox.

🎓 Join AIgenteur Launchpad (free) — community for freelancers and entrepreneurs putting AI agents to work. Want a model tested? Post your request in the community.

🤖 Sister project: AIgenteur — open-source AI cofounder for solopreneurs. ClawEval grades the agents it dispatches.

💡 You'd be surprised how well some small, fast models handle sub-agent tasks — even on an RTX 3090 (we got ours for $799). Before you rent cloud GPUs, check the scores below.

Why ClawEval?

Most benchmarks tell you a model is "smart." ClawEval tells you if it can do the work — route tickets, review code, analyze financials, draft legal docs, plan sprints, and 54 more agent roles. Each test has an exact expected answer. No LLM-as-judge. No vibes.

Built for agent frameworks like OpenClaw and Hermes Agent — any system where you need to pick the right LLM backend for specialized sub-agent roles. If your framework is model-agnostic, ClawEval shows you exactly which model to plug in for each job.

⚡ NEW: Token Efficiency Leaderboard

🏆 Which models waste the fewest tokens? →

Two models both score 10/10 — but one used 500 tokens, the other 5,000. The efficient one saves 10× on compute, cost, and latency. We rank 35 models by tokens-per-point. Think modes use 10–20× more tokens for marginal gains. This metric is hardware-independent.

🚀 NEW: ClawEval v2 — 59 Agents, 1,220 Checkpoints, Zero Ceiling Effects

🏆 The definitive dense evaluation →

Phase F scored x/10 — everyone got 8+. ClawEval v2 upgrades all 59 agents to 15–30 checkpoints each with adversarial traps, sarcasm, and near-truths. Real separation, real rankings. First results (Kimi K2.6, GLM-5.1) dropping now.

🖥️ LOCAL Models — Run on YOUR Hardware

📄 OpenClaw Backend: Local on 3090 — The Complete Guide (PDF) — An interactive walkthrough of running the full OpenClaw backend locally on an RTX 3090. Covers setup, model selection, and performance tips with step-by-step calls to action.

We test quantized open-source models that fit on hardware you already own. Find out which model is best for each agent role before you commit your VRAM. We're also testing smaller models for max context and usability — even a 16GB GPU can run capable sub-agents.

📖 VRAM Guides: 8–16GB Small Models · 16GB · 24GB · 32GB · 48GB · 64GB · 96GB — Which models fit, context limits, speed estimates

🧩 Which Tested Models Fit on Your Hardware?

No discrete GPU? Unified-memory devices can run these models directly. Estimates are conservative — they account for OS overhead (~4–8 GB) and KV cache for reasonable context windows.

⚠️ tight = model loads but leaves little room for KV cache / long context. Short prompts only.

Strix Halo = AMD Ryzen AI Max (128 GB LPDDR5X total, up to 96 GB allocatable to GPU). DGX Spark = NVIDIA GB10 (128 GB LPDDR5X unified, ~96 GB GPU-usable). Both can run 120B-class models with room for KV cache.

Mac Mini M4 Pro 64 GB and M4 Max 128 GB configs exist but are not listed — they slot between the columns above.

☁️ CLOUD Models — Run via API

Don't have a GPU? We also test open-source models hosted on cloud providers so you can compare local vs cloud performance at every agent role. Same tests, same scoring — find out if paying for cloud is worth it, or if your local setup already matches.

💡 Kimi K2.5 leads Phase F at 473/590 (80%), closely followed by M2.7 and Qwen3.5-Plus. M2.7's reasoning tokens count against max_tokens, requiring 16-24k to avoid truncation.

📊 Detailed cloud comparisons:

• ☁️ Full cloud model comparison (all scores per role) →
• 🆕 Phase G — Discriminator Tests (harder tests) →

⚡ Token Efficiency — Which Models Waste the Fewest Tokens?

Two models both score 10/10 — but one used 500 tokens, the other 5,000. The efficient one saves 10× on compute, cost, and latency. This is hardware-independent: it measures model behavior, not GPU speed.

💡 Key insight: Thinking models use 10–20× more tokens than NoThink variants for similar scores. For cost-sensitive agent pipelines, NoThink wins massively on efficiency.

📊 Full Token Efficiency Leaderboard (35 models) → — complete rankings, Think vs NoThink gaps, best per VRAM tier

---

🚀 ClawEval v2 — Full 59-Agent Dense Evaluation

The next generation of ClawEval testing. Every agent role now has a dense constraint test with 15–30 checkpoints — no more x/10 ceiling effects. This is the definitive leaderboard.

Phase F gave every model 8–10/10 on most roles. ClawEval v2 replaces that with granular percentage scoring across all 59 agents. Same roles, dramatically harder prompts, real separation between models.

🏅 ClawEval v2 Leaderboard — Open-Weight Models

📋 Gemini-3.5-Flash — Full 59-Agent Breakdown

Per-role performance across all 59 ClawEval v2 agents. 978/1,220 checkpoints (80.2%) — 22 roles at 100%, 12 roles below 50%.

---

📦 Evaluation Phases

ClawEval evaluates models across multiple phases of increasing difficulty:

Phase F — 59 Roles Across 5 Tiers

• Tier 1 (8 roles): Router, Validator, Health Monitor, Notification, Sentiment, FAQ, Translation, Calendar
• Tier 2 (27 roles): Research, Writer, Editor, Email, Summarization, Customer Support, Data Analysis, etc.
• Tier 3 (11 roles): Code Gen, Code Review, QA Testing, Task Planning, Fact-Checking, Market Research, etc.
• Tier 4 (3 roles): Math/Logic Reasoning, STEM Analysis, Algorithm Exploration
• Tier 5 (10 roles): Orchestrator, Architect, Debugger, Legal, Medical, Financial, Security, SRE, etc.

Scoring Types

exact_json · json_numeric · keyword_detection · code_exec · constraint_check · error_count · action_items · news_dedup · recipe_scaling · compliance_issues · architecture_constraints · manual_review · and more

🚀 Quick Start

Prerequisites

python3 -m venv .venv
source .venv/bin/activate
pip install requests

Run Phase E (12 killer tests)

# Against llama.cpp server
python eval/run_phase_e.py \
  --base-url http://localhost:8080/v1 \
  --model my-model-name \
  --max-tokens 4000

# Against SGLang server (with thinking)
python eval/run_phase_e.py \
  --base-url http://192.168.1.2:8000/v1 \
  --model Qwen3.5-122B-think \
  --api-model /path/to/weights \
  --thinking-budget 16384 \
  --max-tokens 32000

# Disable thinking
python eval/run_phase_e.py \
  --base-url http://192.168.1.2:8000/v1 \
  --model Qwen3.5-122B-nothink \
  --api-model /path/to/weights \
  --nothink

Run Phase F (59 role tests)

python eval/run_phase_f.py \
  --base-url http://localhost:8080/v1 \
  --model my-model-name \
  --max-tokens 4000

# Run specific tests or tiers
python eval/run_phase_f.py ... --test-ids 1 2 3
python eval/run_phase_f.py ... --tier 3

Results

Results are saved to eval/test_results/<model-name>/phase_e/ and phase_f/ directories, including:

• Individual response text files per test
• phase_e_scores.json / phase_f_scores.json with full scoring breakdown

🔧 Key Features

• OpenAI-compatible API: Works with any server exposing /v1/chat/completions (llama.cpp, SGLang, vLLM, etc.)
• SGLang thinking control: --thinking-budget and --nothink flags for reasoning token management
• Think-tag stripping: Automatically strips <think> tags from responses for clean scoring
• Flexible JSON scoring: Case-insensitive keys, nested dict traversal, numeric tolerance
• Deterministic: Every test has an exact expected answer — no LLM-as-judge

📁 Repo Structure

ClawEval/
├── eval/
│   ├── run_phase_e.py          # Phase E runner (12 killer tests)
│   ├── run_phase_f.py          # Phase F runner (59 role tests)
│   ├── phase_e_prompts.py      # Phase E test definitions
│   ├── phase_f/                # Phase F test definitions (5 tiers)
│   │   ├── __init__.py
│   │   ├── tier1.py ... tier5.py
│   ├── role_prompts.py         # 59 role system prompts
│   ├── hard_prompts.py         # Phase D adversarial prompts
│   └── test_results/           # All model evaluation results
├── docs/                       # VRAM tier guides & model selection
│   ├── 16GB, 24GB, 32GB, 48GB, 64GB, 96GB tier guides
│   └── Subagent type reference
├── RESULTS.md                  # Detailed per-role score comparison
└── README.md

🗺️ Model Testing Roadmap

This is a living benchmark. We're continuously adding new models as they release. Here's what's on the radar — and we take requests.

16GB VRAM tier (coming soon)

• Qwen3-8B, Qwen3.5-14B
• Phi-4 14B, Phi-4-mini 3.8B
• Mistral Small 3.1 24B (tight fit)
• GPT-OSS-20B MXFP4
• Gemma 3 12B
• Ministral 3B (ultra-fast routing/triage)

24GB VRAM tier

• ✅ Qwen3.5-35B-A3B Q4_K_M (tested)
• ✅ Qwen3.5-27B Q4_K_M (tested)
• Gemma 3 27B QAT
• More MoE models as they release

64–96GB VRAM tier

• ✅ Qwen3.5-122B-A10B NVFP4 (tested)
• ✅ GPT-OSS-120B GGUF · llama.cpp (tested — 3 reasoning levels)
• More large models coming

Cloud (open-source via API)

• Open-source models via OpenRouter, Alibaba, and other affordable providers

💬 Want us to test a specific model? Open an issue or drop a comment — we prioritize community requests. New models are added as they release.

📄 License

MIT

🔗 Links

• Newsletter: AIgenteur Dispatch — subscribe for new benchmark results and AI agent guides
• Community: AIgenteur Launchpad (free) — discuss results, get help, share what's working
• Sister project: AIgenteur — main cofounder repo. ClawEval is the eval harness it grades itself with.
• GitHub: github.com/AIgenteur/ClawEval