llama-mindcontrol

A llama.cpp fork that extends the reasoning-budget sampler with staged, in-context budget signaling.

Problem

"But, wait..."

Reasoning models generate an unbounded <think> block prior to their final answer, and the length and quality of that block are sensitive to sampling parameters. At the temperature and top-p/top-k settings needed to avoid degenerate, low-entropy output elsewhere in generation, the reasoning block is prone to failure modes that are distinct from ordinary sampling artifacts:

• Repetition loops — the sampler re-enters a previously visited distribution over reasoning tokens, producing near-identical spans of text with no new information content.
• Non-convergent revision — the model repeatedly re-opens a conclusion it has already reached (recurring "wait, actually..." / "but hold on..." transitions) without a stopping condition ever becoming more probable than continuing.
• Unbounded length — absent an explicit stopping signal, nothing in the token distribution guarantees termination; the reasoning block can consume arbitrary context budget before (or without) closing.

The standard/naive mitigation is a hard token-count cutoff enforced by the sampler: once N tokens have been generated inside the reasoning block, </think> is forced regardless of position in the sequence. This bounds worst-case length but does not address any of the above — the cutoff has no dependency on the model's actual generation state and truncates at whatever token index it happens to hit, including mid-token-sequence for a partial word, mid-clause, or mid-computation. It suppresses the symptom (unbounded length) without altering the sampling behavior that produces the loop or the non-convergence in the first place.

Mechanism

This fork adds a state machine around the existing hard-cutoff sampler, with two additional stages that inject fixed text into the reasoning stream at defined points:

1. Intro stage — on entry to the reasoning block, a fixed message stating the token budget is inserted (templated via a {budget} placeholder), e.g. "I'm allowed to think for 512 tokens, so my reasoning should be concise. Let me start by". This gives the model an explicit, in-context reference for its own generation length before reasoning begins.
2. Soft-warning stage — at a configurable fraction of the budget (default 0.5), the sampler waits for the next newline boundary and inserts a fixed message indicating the budget is half-consumed, e.g. "I've used up half of my thinking budget, let me start working towards a conclusion".
3. Hard-stop stage with grace period — once the budget is exhausted, the sampler enters a pending state and waits up to a configurable number of grace tokens for a paragraph boundary (two consecutive newlines) before inserting a fixed closing message and terminating the reasoning block. If no paragraph boundary occurs within the grace period, the cutoff is forced immediately. Total output length remains bounded by budget + grace_tokens in all cases.

Injected text is only inserted at newline or paragraph boundaries, not mid-token or mid-sentence. If the model emits its own </think> before a forced stage would trigger, the natural close takes precedence.

Each stage is opt-in and independently configurable via LLAMA_ARG_THINK_BUDGET_* environment variables at server startup, or as per-request overrides in the API call itself:

Default values preserve upstream's existing hard-cutoff behavior; the new stages are disabled unless configured.

Planned follow-up work: generalize this mechanism into a configurable reasoning template/grammar, rather than a fixed set of budget-based checkpoints.

Benchmarking & Findings

Setup

All results below use Qwen3.6-27B, UD-Q4_K_XL quantization, with MTP speculative decoding (3 draft tokens). A separate pass without speculative decoding produced consistent results and is omitted here for brevity.

Four configurations are compared at several reasoning budgets, each adding one more piece of the mechanism on top of the last:

• Naive — llama.cpp's existing default: the moment the budget is reached, </think> is force-injected immediately, with no grace period and no in-context signaling of any kind. This is the behavior described in the Problem section above, and the baseline this fork is trying to improve on.
• Hard-limit only — this fork's hard-stop-with-grace-period stage used on its own (no soft warning, no intro message): instead of an immediate cutoff, the sampler waits up to grace_tokens for a paragraph boundary before closing the block.
• Soft + hard — the grace-period hard stop plus the soft-warning stage (fired at a configurable fraction of the budget), without the intro stage.
• Intro + soft + hard — the full three-stage mechanism: intro message, soft warning, hard stop with grace period.

Two further reference points appear in the charts and tables: Baseline (unlimited), where the reasoning block runs to its own natural </think>, and — for LiveCodeBench only — No reasoning, where the reasoning block is disabled entirely.

Benchmarks: HumanEval+ (n=164) and LiveCodeBench (release_v6, n=200). Reported token counts are average total completion tokens per test (reasoning block plus final answer), matching the chart axes.

Results: HumanEval+

Two results stand out here:

1. Token consumption drops monotonically as guidance is added, at every budget. Naive uses the most completion tokens of the four configurations at all four budgets, hard-limit-only is next, and soft + hard / intro + soft + hard are consistently the lowest — e.g. at budget 500: 749 (naive) → 672 (hard-limit only) → 592 (soft + hard) → 569 (intro + soft + hard). Soft + hard and intro + soft + hard are close to each other throughout, and which of the two is marginally lower varies by budget on this benchmark (soft + hard is lowest at 750 and 1250; intro + soft + hard is lowest at 300 and 500) — with n=164 this is likely within run-to-run noise rather than a real ordering between the two.
2. Most configurations meet or beat the unlimited baseline (92.7%). Of the 16 budget/configuration combinations, 12 are at or above 92.7%, and 8 exceed it outright — including the best result in the table, intro + soft + hard at budget 1250 (95.7%, using 1360 tokens against the baseline's 2776). The most plausible explanation is that constraining and guiding the reasoning block suppresses the repetition loops and non-convergent revision described in the Problem section above — on a benchmark like HumanEval+, where the model can typically reach a correct answer well within a modest token budget, an unconstrained reasoning block gives the model more opportunity to talk itself into a worse answer, not a better one.

Results: LiveCodeBench

LiveCodeBench is far more reasoning-intensive at baseline (36293 tokens/task on average, versus 2776 for HumanEval+), and the ordering seen above holds even more cleanly here: naive > hard-limit only > soft + hard > intro + soft + hard in total token count, at every single budget tested, with no exceptions. At the 4000-token budget, naive uses 16324 tokens for 70.5% pass@1, while intro + soft + hard uses 7693 tokens — 47% of naive's token count — for 69.5%, a 1-point difference well within what n=200 sampling noise would produce.

Accuracy differences between configurations at a fixed budget are generally small (a few points, consistent with n=200 noise) and don't show a systematic penalty for the more guided configurations — in most cases they hold accuracy roughly level with naive while using a fraction of the tokens.

A separate effect shows up in how each configuration's accuracy responds to increasing the budget. For soft + hard and intro + soft + hard, pass@1 rises monotonically as budget increases from 500 to 4000, with no reversals. Naive and hard-limit-only do not show this: naive drops from 64.0% (1000 tokens) to 62.0% (1750 tokens) before jumping to 70.5% (4000 tokens), and hard-limit-only drops from 65.5% (1000 tokens) to 62.5% (1750 tokens). The guided configurations turn additional budget into a predictable accuracy gain; naive and hard-limit-only do not — this is the clearest "reduced noise" effect in this data.

None of the four budget-constrained configurations fully recovers the unlimited baseline's 72.0% at any tested budget.

By difficulty (LiveCodeBench)

This breakdown clarifies where the token savings come from, and echoes the HumanEval+ result above. On easy problems, every budget-constrained configuration at every tested budget scores at or above the unlimited baseline (96–98% vs. 85%) — again consistent with a capped, guided reasoning block reducing the chance the model overthinks its way into a wrong answer on a problem it could already solve. Medium problems are roughly flat to slightly improved at the higher budget. Hard problems are the exception: accuracy stays well below the unlimited baseline at both the smallest (26–36% vs. 64%) and largest (42–50% vs. 64%) budgets tested, for every configuration including intro + soft + hard. Budget-based control, however it's implemented, does not close this gap — the hardest problems still lose accuracy when reasoning length is capped.

Summary

• Naive (llama.cpp's existing immediate-cutoff behavior) uses the most completion tokens of the four configurations at every budget tested, on both benchmarks — this is the behavior the mechanism is designed to improve on.
• Each additional stage of budget-aware guidance (grace period → soft warning → intro message) reduces token consumption further. On LiveCodeBench this ordering is exact at every budget: naive > hard-limit only > soft + hard > intro + soft + hard.
• Aggregate pass@1 does not show a systematic drop from budget constraints. On HumanEval+, 12 of 16 tested combinations meet or exceed the 92.7% unlimited baseline, and the single best result in either benchmark (95.7%) comes from the most heavily guided, budget-constrained configuration.
• Soft + hard and intro + soft + hard produce a monotonic, predictable accuracy/budget relationship on LiveCodeBench; naive and hard-limit-only do not.
• The gains are not evenly distributed across problem difficulty: easy-problem accuracy improves under constrained, guided budgets (consistent with reduced overthinking), while hard-problem accuracy remains below the unlimited baseline at every budget tested, for every configuration.

Quick start

Configuration is set via LLAMA_ARG_THINK_BUDGET_* environment variables, and can be overridden per-request in the API call — see server API docs for the request-level parameters.

Apple Silicon

Docker on macOS cannot pass the GPU through to a container, so there is no Metal-accelerated Docker image. Build natively instead, following upstream's build guide (Metal is enabled by default on Apple Silicon):

git clone https://github.com/laurencehardman/llama-mindcontrol
cd llama-mindcontrol
cmake -B build
cmake --build build --config Release -j

LLAMA_ARG_THINK_BUDGET="350" \
LLAMA_ARG_THINK_BUDGET_SOFT_RATIO="0.7" \
LLAMA_ARG_THINK_BUDGET_GRACE_TOKENS="64" \
./build/bin/llama-server -m /path/to/your-model.gguf

AMD64 + NVIDIA CUDA

A pre-built Docker image is provided. Example docker-compose.yml:

services:
  llama-server:
    image: ghcr.io/laurencehardman/llama-mindcontrol:cuda
    gpus: all
    ports:
      - "8080:8080"
    volumes:
      - ${MODEL_DIR:-./models}:/models:ro
    environment:
      LLAMA_ARG_THINK_BUDGET: "350"
      LLAMA_ARG_THINK_BUDGET_INTRO_MESSAGE: " I have {budget} tokens to reason through this - that's enough room to work through it carefully, so I'll think it through step by step rather than rushing to a conclusion."
      LLAMA_ARG_THINK_BUDGET_MESSAGE: " [!!NOTE TO SELF] I've used all of my thinking budget, I am now going to wrap up and provide the user their answer."
      LLAMA_ARG_THINK_BUDGET_SOFT_RATIO: "0.7"
      LLAMA_ARG_THINK_BUDGET_SOFT_MESSAGE: " [!NOTE TO SELF] I'm partway through my budget - I should start consolidating toward an answer, but I still have room to finish the important points."
    command:
      - "--model"
      - "/models/your-model.gguf"

MODEL_DIR=/path/to/models docker compose up

Requires the nvidia-container-toolkit on the host.

llama-server exposes the standard OpenAI-compatible API at http://localhost:8080. See the upstream documentation below for other configuration options.

Test it out:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "stream": true,
    "messages": [
      {"role": "user", "content": "Explain how Flash Attention works."}
    ],

    "reasoning_budget_tokens": 350,
    "reasoning_budget_message": "I have reached my reasoning budget - I have enough here to answer now.",

    "reasoning_budget_soft_ratio": 0.7,
    "reasoning_budget_soft_message": "I am partway through my budget - I should start consolidating toward an answer, but I still have room to finish the important points.",

    "reasoning_budget_intro_message": "I have {budget} tokens to reason through this - that is enough room to work through it carefully, so I will think it through step by step rather than rushing to a conclusion.",

    "reasoning_budget_grace_tokens": 50,

    "reasoning_control": true
  }'

---

llama.cpp

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
• subprocess.h - Single-header process launching solution for C and C++ - Public domain