Qwen3.8-Flash-Next and Qwen3.8-27B on one Radeon AI PRO R9700: notes and results

Measurements, settings and lessons from running two local models on a single 32 GB AMD card with llama.cpp and stew675's RDNA4 patch set, as workers for a coding agent (Hermes Agent). Everything was measured between 2026-09-28 and 2026-10-04 on one machine.

These are results for these quants on my rig, not a model ranking. Read Caveats before quoting a number.

TL;DR

• Flash-Next (a 125B mixture-of-experts model) runs usefully on one 32 GB card plus 64 GB of RAM. With the AtomicChat Q4_K_M quant and patch release r30: two workers with 131,072 tokens of context each, 36 tok/s for one worker and 50 tok/s combined for two, without a draft head.
• On r29/r30, set GGML_SCHED_DEVGATHER=0 or the output is garbage. With the default, the first request after loading is correct and every later one is ////////, at full speed. Seen on all three quants tested. Upstream knows (issue #85) and says r31 changed the default; nothing newer than r30 was tested here.
• Load with --lazy-mode on --load-mode none. mmap and dio ran a 61 GiB machine out of RAM.
• Check the text before you measure speed. The corrupted server was first reported here as "48 tok/s, pass".
• The expert cache is what makes it fast (about 39 tok/s against 17-20 without it). The draft head adds 27-45% for one request once the cache is on, but costs VRAM, pushes more of the model into RAM, and left only about 1.7 GB of RAM available under a real agent workload.
• Tool calling: level on ordinary tasks, Flash-Next ahead on hard ones. tool-eval-bench main suite 94.8 against 95.3 (27B) on the scenarios graded for both; Hard Mode 93.9 against 79.8, three runs each.
• The 27B is the faster model: about 1.7 times as fast per turn in the benchmark, reads prompts 1.6 to 2 times as fast, and has working vision here.
• A 220 W power cap costs almost nothing: 78 °C instead of 87-89 °C, prompt reading about 3% slower.

Details and the reasoning: FINDINGS.md. All tables: results/.

Hardware

Headless, used over SSH. One GPU; the CPU's built-in graphics is hidden from ROCm with HIP_VISIBLE_DEVICES=0.

Software versions

The patch repo is rebased often. On 2026-10-04 its head was v16-a55e952b8-r10, on a newer llama.cpp base. To reproduce these numbers, use the patch repo at commit f108261 (r30) or 72976d8 (r20).

Model files

The AtomicChat files carry an older chat template; it is run with the standard one (--chat-template-file, extracted from the ISTA file). None of the Flash-Next files contains a draft head.

Results

All speeds are tokens per second at medium reasoning effort, temperature 1.0, top_p 0.95, top_k 20, min_p 0, on a short prompt unless said, with the text checked first. Speeds fall as the context fills.

The options

The 27B's range is wide because its speed follows draft acceptance. The other three 27B options (27b-q5xl-1agent, 27b-q6xl, 27b-q4xl) were measured for memory only: results/options-and-memory.md. Under a real agent workload flash-next-q4-2x128k fell to about 1.7 GB of available RAM, against 7.8 GB in the test.

More: Stage 1 (ISTA quant, every configuration tried, loading modes), Stage 1d (three quants, the draft head), Stage 1e (2 x 131,072 and 1 x 262,144).

Tool calling (tool-eval-bench 2.7.0)

flash-next-q4-2x128k-plain against 27b-q5xl, identical settings, seeds 42, 43 and 44.

In Hard Mode the 27B lost TC-74 and TC-84 in all three runs by sending a tool call that depended on an earlier call's result in the same turn. Four main-suite scenarios (TC-65, 66, 67, 69) were rejected by both servers with a llama.cpp grammar error and are in neither score; TC-45 was graded on one server and not the other. Details: results/tool-eval-main.md, results/tool-eval-hard-mode.md, raw reports in results/raw/tool-eval/.

One real agent job on each model

A small game built as four delegated tasks, two workers at a time, checked by a cloud parent model.

One run each. results/stage3-hermes-job.md

How to reproduce

1. Driver stack. Ubuntu 24.04.4 with kernel 6.17.0-42 pinned, amdgpu-dkms, ROCm 10.0.0. The traps are in FINDINGS.md.

2. Build llama.cpp with the patch set (r30 shown; the patch repo's release.json names the base commit):

   git clone https://github.com/stew675/llama-cpp-rdna-boosts ~/llama-cpp-rdna-boosts
   git -C ~/llama-cpp-rdna-boosts checkout f108261              # r30; 72976d8 for r20
   git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp-rdna-r30
   cd ~/llama.cpp-rdna-r30
   git checkout "$(jq -r .base ~/llama-cpp-rdna-boosts/release.json)"     # 84e76d8a2
   git config user.name "Your Name"; git config user.email "you@example.com"   # git am needs an identity
   bash ~/llama-cpp-rdna-boosts/scripts/apply-all.sh .          # 16 commits, "rdna-boosts: block 00..15"
   git rev-parse HEAD^{tree}                                    # must equal .tree in release.json
   HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
     cmake -B build -DGGML_HIP=ON -DGPU_TARGETS="gfx1201" -DCMAKE_BUILD_TYPE=Release
   cmake --build build -j6
   

3. Start a server. Each file in configs/ is one setup; scripts/start-server.sh turns it into a llama-server command. The everyday Flash-Next setup, written out:

   cd ~/llama.cpp-rdna-r30
   MOE_EXPERT_CACHE_MIB=4096 MOE_EXPERT_CACHE_DEVMAP=1 GGML_SCHED_DEVGATHER=0 HIP_VISIBLE_DEVICES=0 \
   ./build/bin/llama-server \
     -m ~/models/flash-next-atomic/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
     -ngl 99 -ncmoe 41 -c 262144 -np 2 -t 6 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
     --lazy-mode on --load-mode none --cache-ram 2048 \
     --chat-template-file ~/models/flash-next/chat-template-standard.jinja \
     --min-p 0 --metrics --alias qwen3.8_local --host YOUR-ADDRESS --port 8080
   

4. Check the text first. python3 scripts/text-check.py http://YOUR-ADDRESS:8080 qwen3.8_local NAME sends three fixed prompts and fails on a wrong answer or on runs like ////. Send more than one request: a corrupted server answers the first one correctly.

5. Tool-calling benchmark.

   uv tool install --python 3.12 "git+https://github.com/SeraphimSerapis/tool-eval-bench.git@v2.7.0"
   tool-eval-bench run --base-url http://YOUR-ADDRESS:8080 --model qwen3.8_local --backend llamacpp \
     --seed 42 --temperature 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
     --backend-kwargs '{"reasoning_effort":"medium"}' --timeout 600 --no-live
   

   Add --hardmode-only for Hard Mode, or --context-pressure 0.75 --context-size 85248 for the pressure run. scripts/teb.sh runs the whole sequence in a detached tmux session.

What is in this repo

Caveats

• One machine, one card. Nothing here was repeated on other hardware, other drivers or other patch releases.
• Medium reasoning effort and temperature 1.0 throughout, because that is what the agent gets: Hermes sends "reasoning_effort":"medium" and no sampling settings, so the server's defaults apply (temperature 1.0, top_p 0.95, top_k 20, and min_p 0 from the option files). The chat template's own default is xhigh, and tool-eval-bench's default is temperature 0, so numbers from elsewhere may not be comparable. ISTA say their quant was calibrated at xhigh and loses quality at medium.
• Run counts are small. Speeds: three runs on the short prompt, one at each larger size. Main benchmark and Hard Mode: three runs per model. Context pressure, IFEval and the real agent job: one run per model.
• These quants on this rig, not a model ranking. A 4.27-bit Flash-Next quant is compared with a Q5_K_XL 27B quant because those are what fit. Different quants, engines or settings can change the order.
• The benchmarked Flash-Next setup had no draft head. It is assumed that drafting changes speed and not answers; that was not tested. The real agent job did use the draft head.
• The two servers are different patch releases (r20 for the 27B, r30 for Flash-Next). The 27B was not tested on r30.
• Sampled, not greedy. At temperature 1.0 runs differ; the three seeds show by how much.
• Estimates are marked as estimates in the tables. Everything else was measured.
• Vision was not tested on Flash-Next. The 27B's vision works on this rig.

Licence

MIT, see LICENSE. The files under results/raw/tool-eval/ were produced by tool-eval-bench (MIT) and contain its scenario texts.