clef-langchain-bench

A local decision model as the router of a LangChain/LangGraph workflow, compared with a classic tool-calling agent.

Why this exists

I'm a third-year student, and the agentic business workflows I'm working on were capable but slow. So for the common paths I kept falling back to scripts with keyword rules, which were fast but brittle: the Turkish word for "stock" shows up both in "buy 100 lots of THY" and in "what is a stock?".

When Cloudflare released the Clef decision models and LangChain wrote about using Jev in agent harnesses, I wanted to check, on hardware I actually have, whether a decision model could sit in between: deterministic code for the requests I already know how to handle, an agent for everything else. This repo is what I found. It is a learning project, not a product claim. If something here is wrong or unfair to one of the approaches, please open an issue; corrections are very welcome.

This is an independent project. I'm not affiliated with Cloudflare, LangChain or TypeSafe.

Where this pattern might help (ideas I haven't benchmarked yet):

• Ticket and email triage: department, urgency and phishing as one typed question set, in one call.
• Choosing the tool or sub-agent before an LLM is ever called, and escalating to the LLM only when confidence is low.
• Guardrails before tool execution: AutoModeMiddleware from langchain-typesafe worked unchanged against the local server.
• Log and alert triage: severity and component on every line, cheap enough to run on a stream.
• Document intake with images, such as "is this receipt legible, which type, does it need approval?" (clef-flash accepts images).

Agentic workflows are flexible but slow, so many teams fall back to hand-written scripts with keyword rules. Those rules are brittle: "hisse" (stock) appears in both "buy 100 lots of THY" and "what is a stock?". This repo measures a middle path:

• Cloudflare/clef-flash (9B decision model) reads the request and returns calibrated probabilities for typed questions (intent, ticker) in a single forward pass. It generates no text.
• Code runs the matching tool deterministically and formats the reply.
• An LLM agent handles only the open-ended requests (or anything clef is unsure about).

Everything runs locally on two consumer/workstation GPUs. Clef speaks the Jev/SystemOne API, so the stock langchain-typesafe client and its middleware work against it unchanged; the only change is setting TYPESAFE_BASE_URL.

Results at a glance

Turkish stock-brokerage workflow, 15 requests × 3 runs, same tools, same LLM (Qwen3.5-9B, thinking off):

• On requests that map to a tool, the clef route was ~27× faster (median 0.27 s vs 7.20 s).
• The agent failed 2 requests in all 3 runs by answering without calling the tool and inventing a price: "Koç Holding şu an kaç lira?" → "185,40 TL" (backend price: 198.75); "Garanti hisselerimi satmayacağım, sadece fiyatına bakacaktım" → "10.45 TL" (backend: 131.90). In the clef route, prices always come from code.
• Both approaches handled the traps correctly: "Hisse senedi nedir?" ("what is a stock?") and "Aselsan alınır mı sence?" ("should I buy Aselsan?") executed no order; "I won't sell, I just wanted the price" became a price lookup in the clef route (p=0.92), where a keyword rule would have matched "sat" (sell).

Full per-request table: results/workflow/summary.md. Raw runs with replies: results/workflow/results.json.

The two graphs

A) Tool-calling agent — langchain.agents.create_agent (exported from the compiled graph). The LLM decides on every step:

graph TD
    start([start]) --> model
    model -.->|tool calls| tools
    tools -.-> model
    model -.->|final answer| finish([end])

B) Clef-routed workflow — a LangGraph StateGraph. One clef call answers two questions; code does the rest:

graph TD
    start([start]) --> classify["classify<br/>clef-flash: intent + ticker<br/>regex: quantity"]
    classify -.->|price| price["price<br/>get_price()"]
    classify -.->|buy| buy["buy<br/>buy_stock()"]
    classify -.->|sell| sell["sell<br/>sell_stock()"]
    classify -.->|portfolio| portfolio["portfolio<br/>get_portfolio()"]
    classify -.->|missing ticker / quantity| ask["ask_missing"]
    classify -.->|other, or p below 0.6| llm["llm_fallback<br/>Qwen3.5-9B"]
    price --> finish([end])
    buy --> finish
    sell --> finish
    portfolio --> finish
    ask --> finish
    llm --> finish

The graphs are generated by the benchmark itself: results/workflow/graph_agent.mmd, results/workflow/graph_clef.mmd.

The single clef call behind classify:

from langchain_typesafe import Choice, TypeSafeClassifier

classifier = TypeSafeClassifier()  # TYPESAFE_BASE_URL=http://127.0.0.1:30000
answers = classifier.invoke({
    "state": "Türk Hava Yollarından 100 lot al",
    "questions": {
        "intent": Choice(instructions="What does the user want the brokerage system to do right now?",
                         criteria={"price": "...", "buy": "...", "sell": "...", "portfolio": "...", "other": "..."}),
        "ticker": Choice(instructions="Which stock does the message refer to?",
                         criteria={"THYAO": "Türk Hava Yolları", "GARAN": "Garanti BBVA", ..., "none": "No specific stock"}),
    },
}).choices
# answers["intent"].choice == "buy" (p=0.94), answers["ticker"].choice == "THYAO"

Setup

graph LR
    subgraph client["Client: Apple M4, 16 GB (macOS)"]
        bench["workflow_bench.py<br/>LangChain / LangGraph"]
    end
    subgraph server["GPU server: Windows 10, 2× RTX A5000"]
        api["clef_server.py (FastAPI, 127.0.0.1:30000)"]
        gpu0["GPU0: clef-flash<br/>POST /v1/systemone"]
        gpu1["GPU1: Qwen3.5-9B<br/>POST /v1/chat/completions"]
        api --- gpu0
        api --- gpu1
    end
    bench -->|"SSH tunnel over home Wi-Fi"| api

flash-linear-attention and causal-conv1d were not installed (they need a CUDA toolchain on Windows), so both Qwen3.5-based models ran on transformers' pure-PyTorch fallback. That slows both sides; Qwen decoded at ~12–17 tokens/s.

server/clef_server.py exposes:

• POST /v1/systemone — Jev/SystemOne-compatible decisions (clef-flash), usable by TypeSafeClassifier, ModelRouterMiddleware, AutoModeMiddleware.
• POST /v1/chat/completions — OpenAI-compatible chat with tool calling (Qwen3.5-9B). Model names: qwen-fast (thinking off), qwen-think (thinking on; reasoning is force-closed at its token budget so it cannot loop forever).

Second benchmark: decision model vs. generative LLM on classification

10 short classification cases (support routing, sarcasm, phishing vs. a legitimate GitHub email, a Turkish complaint, news topic, banking intent, toxicity, NLI, log severity), 15 questions in total:

• clef-flash missed only sarcasm detection (p=0.45 for "sarcastic").
• clef-omni does not fit two 24 GB cards in BF16, so server/clef_omni_loader.py loads it with device_map="auto", moves the joint-schema head's inputs to the head's device, and reloads lm_head from safetensors when accelerate offloads it to the meta device. Copying offloaded layers on every forward pass explains the 5.8 s.
• clef-omni also answered from audio only (two synthesized voicemails) and from images; see results/classification/omni.log.

Results: results/classification/. Note: the summary line in llm.log reports 25/30 for thinking mode; that was a bug in the summary filter (now fixed in server/bench_llm_baseline.py). The per-case lines and llm_results.json give the correct 10/15.

Reproduce

GPU server (Windows, NVIDIA GPUs with ≥ 20 GB each):

pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements-server.txt
:: copy server\*.py to %USERPROFILE%\clef, then download the models once:
python -c "from huggingface_hub import snapshot_download as s; s('Cloudflare/clef-flash'); s('Qwen/Qwen3.5-9B')"

Windows OpenSSH Server must be enabled. The server listens on 127.0.0.1 only and is reached through the SSH tunnel.

Client (macOS/Linux):

python3.12 -m venv .venv && .venv/bin/pip install -r requirements-client.txt
export CLEF_HOST=user@your-gpu-box
scripts/start_server.sh        # terminal 1: tunnel + server, Ctrl+C stops both
.venv/bin/python client/workflow_bench.py --repeats 3   # terminal 2

The classification benchmark runs on the GPU server itself: python bench_classification.py (clef-flash, then clef-omni) and python bench_llm_baseline.py (Qwen3.5-9B).

Limitations

• Small, hand-written test sets (15 + 15 questions) written by the author, Turkish requests in the workflow benchmark. Read the results as a demonstration of the pattern, not as a general accuracy claim. See the Decision Index for broad evaluations.
• The clef route only covers predefined workflows. Adding a workflow means writing code, and clef cannot extract free-form values: quantities come from a regex here, a long ticker list would need narrowing first.
• Templated replies. The clef route answers with fixed strings; the agent writes more natural text.
• One machine, no kernel optimizations, Wi-Fi in the loop. Absolute numbers will differ on other hardware or with an optimized server such as SGLang; the relative gap comes from one forward pass vs. several generation calls.
• The agent uses a 9B open model. A frontier model would likely not invent the prices, but would add API latency and cost to every step.

License

MIT for the code in this repository. The models have their own licenses: clef-flash, clef-omni and Qwen3.5-9B are Apache-2.0.