I've had a Strix Halo board for about a year and been playing around with it for various projects when it's not just being a beefy linux machine. I ordered the 128GB Framework Desktop board pre-panic and I'm very grateful for that. I also grabbed an R9700 Pro AI card late last year for another machine, thinking it would be fun to compare the two. I ended up parting out the machine the R9700 was in for something else and wondered what might be possible with the R9700 in the Strix Halo machine. On the Framework desktop board, there's an x4 4.0 slot hanging out. I already had an x4 extension cable so I could mount a 25G card in it, but a GPU would fit just fine too. I have my board in a Fractal Design case instead of the framework shell (bought the bare board), so I had plenty of room for the card and my power supply had the new 12V connector. Even with today's pricing, a Framework Strix Halo 128GB board and an R9700 is about ~5k all in, so similar price to a DGX spark but with a little more RAM (~160GB, obv with caveats), and it's a regular 16-core ryzen PC instead of the tacky gold box.

So, the kicker is that it works. 49 tok/s, 682 tok/s prefill at 32K - double the stock 24 tok/s and 2.5x prefill on Qwen-3.5-122b.

Getting this working was a little bit of a mind-bender, so wanted to share with people. Here's how it works. We can't just slap part of the model on the R9700 and expect it to be good though. It's actually worse if you try to do that in most cases. First, we need to place the parts of the model that benefit from the different parts of the hardware. So, with a big MoE model like this, we have a bunch of data that only gets touched for some tokens and those routed experts need to get put on the Strix in the bigger unified memory pool. It works out to about 62GB of the 71GB model, but we might only read 2GB of it per token. The dense parts of the model are about ~4GB and since they get touched for every token, we can put that on the R9700 where we have more compute and memory bandwidth. So we put KV cache, the dense part of the model, and critically, the MTP drafter on the R9700. We can stuff the remaining VRAM on the R9700 with as many layers as fix, which in my setup was 14. This all works because only about 12KB of data per token needs to cross that narrow x4 4.0 link, so as long as the latency isn't bad, it doesn't matter. Trying to do something like Tensor Parallelism across these two would not work well because of that bottleneck.

Here's part of the config:

llama-server -m Qwen3.5-122B-A10B-Opus-Reasoning-Q4_K_XL.gguf
-dev ROCm0,ROCm1 -ts 1,0 --fit off -ngl 999 -fa on --jinja --no-mmap
-ot 'blk.(1[4-9]|[2-4][0-9]).ffn_(gate|up|down)_exps=ROCm1'
-c 32768 -ub 4096 -b 4096
-md mtp-draft-out-q4_K.gguf --spec-type draft-mtp -devd ROCm0
--spec-draft-n-max 4 --spec-draft-p-min 0.5

I kept going on tuning, and tried to reduce the number of kernel launches, which seemed to be holding back performance. I wasn't hitting anywhere near the right numbers per the theoretical bandwidth for each device. I made some updates to llama to make this work, linked on github below. The variant of the model I was using is also linked below, which is a fine tune that I requantized and grafted on an MTP head for my use on a different project.

https://huggingface.co/SixVolts/Qwen3.5-122B-A10B-Opus-Reasoning-MTP-GGUF

Qwen3.8-Flash-Next on the same box

New model, who dis. Same idea as above: dense trunk, KV cache and the draft head on the R9700, the routed experts of most layers on the Strix, the n-gram table in host RAM. This repo's main is upstream master plus the MTP work from unslothai/llama.cpp#144 and ggml-org#28118, plus the kernel and scheduler changes in HALO-HYBRID.md. On this layout stock llama.cpp decodes at 27–28 tok/s; this branch does ~45 (52 greedy), and prefills at ~1,500 tok/s.

Launch (single user, 8K context; ROCm0 is the R9700, ROCm1 the iGPU — check the device order in the startup log):

sudo sh -c 'echo 2 > /proc/sys/vm/drop_caches'   # drain the iGPU's TTM pool before a big load

LLAMA_PREFILL_LANES=2 \
llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -dev ROCm0,ROCm1 -ts 1,0 --fit off -fa on -ngl 999 -c 8192 -b 4096 -ub 1024 --no-mmap -np 1 \
  -ot 'blk\.(1[4-9]|[2-4][0-9])\.ffn_(gate|up|down)_exps=ROCm1,per_layer_token_embd=CPU' \
  -md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -devd ROCm0 -ngld 999 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --host 0.0.0.0 --port 8080

• Model: unsloth/Qwen3.8-Flash-Next-GGUF UD-Q4_K_XL (four shards, 111 GB). Draft head: the shared-Q8_0 file in that repo's MTP/ folder (2.6 GB); it borrows the target's embeddings and lm head, so -devd ROCm0 is required.

• Layout: -ot sends the routed experts of layers N–47 to the iGPU ("hybrid-N") and keeps the n-gram table in host RAM. Each layer kept on the R9700 costs it ~1.4 GB, so pick N by what has to fit next to the 2.6 GB head:

  context	layout	-ot pattern
  1 slot, 8K	hybrid-14	blk\.(1[4-9]|[2-4][0-9])
  1–2 slots, 16K each	hybrid-12	blk\.(1[2-9]|[2-4][0-9])
  2 slots, 32K each	hybrid-11	blk\.(1[1-9]|[2-4][0-9])
  64K+ (sparse-attention gather turns on by itself)	hybrid-10	blk\.(1[0-9]|[2-4][0-9])

  Without the head, 4 slots at 32K fit at hybrid-12. -ctk q8_0 -ctv q8_0 halves the KV cost (1.1 GB per 32K slot).

• Prefill: LLAMA_PREFILL_LANES=2 runs consecutive ubatches on two schedulers so the iGPU's expert GEMMs overlap the R9700's attention; with the MoE GEMM tile fixes and the R9700's f32 matmul paths (all on by default) warm prefill at 3.7K / 15K / 30K is 1503 / 1258 / 1025 tok/s at -ub 1024 and 1626 / 1324 / 1049 at -ub 2048, from 676 / 616 / 540 on the single-lane branch. The second lane costs a second set of compute buffers on the R9700 (0.75 GB at -ub 1024, 1.5 GB at 2048), so with the head use -ub 1024; -b must be at least twice -ub. Details and the scheduler fixes it needed: HALO-HYBRID.md, "Two-lane prefill".

• Decode: --spec-draft-n-max 2 (3 is the same within noise, 4 is worse). Acceptance is ~0.70 greedy and ~0.51 with the model-card sampler (temperature 1.0, top-p 0.95, top-k 20), which is the whole difference between 52 and 45 tok/s. The head only pays at one or two streams; for more users leave the -md/--spec-* lines out.

• API: use /v1/chat/completions (a bare prompt on /completion stops after one token with this model). The model thinks by default; "chat_template_kwargs": {"enable_thinking": false} turns it off per request.

• Memory: ~51 GB of experts on the iGPU, the 28.8 GB table plus page cache in host RAM, 23–26 GB plus the head on the R9700. Drain caches before launching after big file activity.

Measured on this build (model-card sampler, 4K prompts, 256-token completions; -b 4096 -ub 1024, LLAMA_PREFILL_LANES=2; "agg" is the sum over streams, single-stream rows are the per-stream number; the prefill column's first request of a fresh server is cold, warm numbers are 10–20% higher):

llama.cpp

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain