cudafast — Qwen 3.8 125B A6B CUDA

This branch (lbf/pq2-rot) is the fast runtime for Qwen3.8-Flash-Next-Mooney: it loads the model's PQ2_0 ternary experts + lowbitflash.rot.* metadata and runs on one DGX Spark with MTP speculative decoding (45.8 tok/s short / 47.7 tok/s at 4k measured). See MOONEY.md for what changed and how to run it — or just use mooney-spark for a one-command setup. Build on GB10 with make -C ds4 cuda-spark CUDA_ARCH=sm_121 -j8. Engine lineage: Layr-Labs/cudafast-qwen38-125b-a6b-engine → vendored Layr-Labs/ds4 → antirez/ds4 (MIT). Compute thanks to Zach Mueller and Lambda.

The original benchmark-repo README follows.

cudafast — Qwen 3.8 125B A6B CUDA

This repository is the engine for the Qwen 3.8 125B A6B CUDA speedup benchmark. The track identifier is qwen3.8-125b-a6b-cuda-v1.

The engine is the ds4 port (Layr-Labs/ds4). The scored engine is the Engine Protocol v1 adapter (harness/protocol-adapter) with the ds4 C/CUDA engine linked in-process. The engine source is VENDORED at ds4/, pinned to one commit of Layr-Labs/ds4, our port of antirez/ds4; tools/ds4/build.sh builds it. The target is the UD-Q4_K_XL GGUF conversion of Qwen 3.8 Flash Next, four shards plus a Q8_0 MTP draft head. The port carries the qwen4exp family the target needs: the multi-shard loader, the qwen4exp kernels and graph, the nextn MTP head behind --mtp-model and the speculative cycle at depths 1 to 6. The participant-editable surface is the vendored engine (ds4/), the engine adapter (harness/) and the MTP head declaration (mtp-head/, mtp-head.manifest.json); benchmark.json editablePaths is the authority. This repository was seeded from the Gemma 4 26B A4B MLX engine and then carried a vLLM engine; both are gone. docs/qwen38-125b-a6b-port-notes.md is the engineering record.

What this repository is

This repository holds the engine. You optimize the engine. You make the model do the same work in less time.

The benchmarker measures the engine. The benchmarker is a separate program called benchd. It arrives as a verified prebuilt binary. It owns all timing, all scoring, and all gates. Nothing in this repository measures or scores anything.

The ranked run is SINGLE-STREAM. It times each pinned prompt in its own window, one at a time, on your engine and on a serial control engine. The score compares the two.

NOTE — official scoring is ARMED. fixtures/qwen3_8_125b_a6b_track.json sets official_scoring_enabled to true. That flag is the single authority on the arm state, and the benchmarker enforces it: it refuses to seal an official scoring artifact while the flag is false or absent. What is still box work, not repository state, is staging the reference workspace and calibrating each ranked box. The launch reference stays stock serial (num_speculative_tokens 0); the native MTP head is the improvement path.

Lineage

This repository descends from Layr-Labs/mlxfast-qwen-38-27b-mtp-engine, which descends from Layr-Labs/mlxfast-challenge-dev. Those repositories rank different models under different rules. Only this track's rules apply here.

A few fixtures and transform validators still carry Qwen or Laguna in their names. Those names point at real foreign checkpoints on purpose. They are the negative controls and the fixture substrate that this track's own gates are tested against.

Requirements

The scored engine is the Engine Protocol v1 adapter with the ds4 engine linked in-process. ./setup.sh builds the engine and the adapter, stages the adapter, and verifies the target snapshot. The steps need different things.

• A CUDA toolkit (nvcc, CUDA 13) and a C compiler. tools/ds4/build.sh builds upstream's cuda-spark target for sm_121 (the GB10) and the shim library the adapter links. This is box work. Off the box, tools/ds4/build.sh --cpu-check proves the pin and the shim agree without CUDA.

• A Rust toolchain (cargo). The adapter builds with --features ds4-engine against the shim library. The default (mock) build is portable and needs no GPU, so the adapter's own tests run on a laptop and in hosted CI.

• The GGUF target snapshot: the four target shards and, flat beside them, the Q8_0 MTP draft head. Every file is verified against the {bytes, sha256} pins in fixtures/qwen3_8_125b_a6b_track.json. Where it lives depends on MLXFAST_TARGET_SNAPSHOT_DIR:

  • Set (a ranked box): the organizer-staged directory. Setup only verifies it, on every run. It fetches nothing, deletes nothing, and FAILS CLOSED when the directory or a file is absent or does not match.
  • Unset (your own machine): reference_weights/Qwen3.8-Flash-Next-GGUF under the checkout. Setup downloads each missing file from the pinned revision of the public model repository (about 114 GB, resumable), keeps a download only when both pins match, and leaves a marker after a verified pass so a rerun there reads no shard.

  Setup never substitutes or re-quantizes a checkpoint.

• jq and Git.

The benchmarker arrives as a prebuilt binary. ./tools/fetch-benchd.sh resolves it from the qwen3.8-125b-a6b-v1 dist channel and verifies it against the channel's own benchd.manifest.json. That channel is the public bench repository Layr-Labs/mlxfast-bench, so the fetch needs no token. The channel holds one binary per platform and the resolver REFUSES a binary whose target_triple is not this host's.

Quickstart

Run these commands in order. One sentence describes each command.

git clone <repository-url> cudafast-qwen38-125b-a6b-engine

This command copies the repository to your machine.

cd cudafast-qwen38-125b-a6b-engine

This command makes the repository your working directory.

./tools/fetch-benchd.sh

This command resolves the benchmarker binary from the dist channel into benchd-bin/ and verifies its sha256, its byte count and its platform against the channel's benchd.manifest.json (installed beside the binary).

./setup.sh

This command builds the ds4 engine and the cuda-engine adapter, stages the adapter at the path benchd resolves, then makes the GGUF target snapshot present and verified. With MLXFAST_TARGET_SNAPSHOT_DIR unset it downloads each missing file from the pinned public model repository into reference_weights/ and checks every file against the contract's {bytes, sha256} pins; a rerun on the same machine reads no shard. With it set, setup only verifies the staged directory. See Requirements for the toolchain each step needs and the environment variables that skip a step.

tools/local-baseline.sh

This command runs the local test: one leg of your engine on the public golden, through ./benchmark.sh --local-iterate, with the cool gate on. It needs no organizer material and it seals no score (score: null is the expected result). It needs the GPU and the snapshot, so it is a box command; off the box it stops before the engine boots.

The adapter's own unit tests run without a GPU or a checkpoint:

cargo test --manifest-path harness/protocol-adapter/Cargo.toml

This command exercises the protocol loop, the verb translation, and every error path against a mock transport. It needs no GPU and no checkpoint.

NOTE — the ranked run is box work. The GPU, the pinned target snapshot and the ranked goldens all live on the box. Off the box the local signals are the adapter tests above, the engine syntax check (tools/ds4/build.sh --cpu-check) and the shell tests (tools/test-*.sh). See "Local testing vs the ranked run".

correctness_prompts/ holds three public files. The two public_longcopy_gate_english_1024_*.json files are Gemma-era captures kept for their 1024-token prompts; they do not load against the Qwen target. public-longcopy-gate-english-1024.golden.json is a public Qwen capture (unsloth/Qwen3.8-Flash-Next-GGUF @ 38bb39ee97821de2c9009abb7e93950eec396e66) for local runs (docs/qwen38-125b-a6b-port-notes.md section 5).

Repository structure

benchmark.json editablePaths lists four entries: ds4/, harness/, mtp-head/ and mtp-head.manifest.json.

The MTP head

This track's MTP head is NATIVE: it is a separate Q8_0 GGUF that the organizer stages in the target snapshot beside the shards (docs/qwen38-125b-a6b-port-notes.md section 6). tools/serve-up.sh loads that file. No submission downloads or carries a head weight.

The head is used exactly as staged. No custom head is accepted: a head declaration accepts "source": "pinned" only.

The head directory

mtp-head/ holds one README.md and nothing else. It is an editable path, and the byte budget bars a weight file in it.

NOTE — keep that README.md in place. It records what the directory is, and it keeps the directory present in a fresh clone.

What you may change

benchmark.json editablePaths is the authority. It lists four entries: ds4/ (the vendored engine), harness/ (the engine adapter), and mtp-head/ plus mtp-head.manifest.json (the MTP head declaration). The rule behind the list is simple. Code that proposes tokens or computes the forward pass is editable. Code that verifies, measures, or ledgers stays trusted — the benchmarker (benchd), the gates, fixtures/ and this manifest.

The engine is yours to change

ds4/ is the inference engine, vendored into this repository as plain files and participant-editable. Change a kernel, the graph, the scheduler, the Makefile flags -- benchmark.json editablePaths is the authority, and it lists ds4.

ds4/VENDOR.json records the signed commit of Layr-Labs/ds4 the tree came from. You do not need that repository: the engine is here.

How to iterate. Edit under ds4/, then rebuild and test:

tools/ds4/build.sh
cargo test --manifest-path harness/protocol-adapter/Cargo.toml

The build cache keys on the engine's CONTENT, so your edit is always rebuilt and never served from a previous build.

What is protected. benchmark.json, benchmark.sh, fixtures/, correctness_prompts/ and tools/ are the contract, the gates and the oracles. Yukon archives and overlays only editablePaths, so a change outside them never reaches the measured tree. Code that proposes tokens or computes the forward pass is editable; code that verifies, measures or ledgers is not.

Your edit must still be correct. The correctness goldens are the gate: an engine change that alters the token stream fails them. Speed is what you are optimizing; the tokens are what you must preserve.

The organizer-pinned head

The speculative head is the organizer's pinned weights. It is used exactly as staged. You may not re-quantize it, replace it, or upload head weights of your own. Nothing about the head is participant-tunable except the draft depth, which you declare in mtp-head.manifest.json.

mtp-head/ is an editable path, but the editable byte budget bars a head weight file: a real head weight far exceeds maxFileBytes (4473321) and is refused before any measurement.

The declaration file stays editable and optional: mtp-head.manifest.json. Its live field is spec, and tools/spec-declaration.sh reads it:

"spec": { "enabled": true, "num_speculative_tokens": 1 }

source must be "pinned", and an unknown top-level key is refused by name; tools/spec-declaration.sh checks both. The only head that can load is the organizer-staged one, so the value selects nothing. max_bytes, bytes and sha256 are recorded, not read; ./setup.sh verifies the head bytes against the contract's own pin instead.

A declared sha256 is optional and the runner does not verify it against the head bytes. The head bytes are bound one level up: the head is part of the pinned target snapshot, and ./setup.sh verifies every staged file against the contract's {bytes, sha256} pins. docs/participant-contract.md section 4 is the authority for the declaration rules.

An absent declaration is serial: the drafter is off. That is the normal case. A declaration that is present but broken is a refusal. The runner never falls back silently.

A head only proposes tokens. The organizer-pinned target model decides every emitted token. The serial control leg always runs with the drafter off.

Batch size and draft depth

NOTE — batch size is locked at 1. Draft depth is not locked. The scored batch size is 1: this track is scored single-stream (David ruling 2026-08-27). You may not tune it.

The draft depth is a free lever, and it is not pinned at 1. You declare it in mtp-head.manifest.json under spec.num_speculative_tokens.

Select a depth from 1 to 6. The pinned engine implements all six (DS4_IMPLEMENTED_DEPTH in harness/protocol-adapter/src/ds4_backend.rs). A depth outside 1 to 6 is refused by name, never clamped (permitted_draft_depths in the track fixture). benchd measures at depth 2 when the invocation names no depth; an mtp block with no depth key resolves to the ceiling of 6.

Every run seals what actually ran: effective_spec for the declared arm and depth, effective_mean_draft_len for the realized draft length.

The byte budget

benchmark.json editableSurfaceByteBudget caps the enforced editable surface.

Every editable path is enforced. Nothing is exempt.

exemptPaths is absent since 2026-08-26. The exemption existed to let head weights ride in a submission outside the source budget. A submission carries no head weights any more, so there is nothing to exempt. The two exempt caps stay declared because both enforcers carry the same numbers as compiled-in fallbacks and this manifest is what holds them to a reviewed value.

The organizer-staged head is not walked by this budget at all. It sits in the target snapshot on the box, which is not an editable path, so the walk never visits it.

The weights are frozen

The target model's quantization is frozen as shipped, and so is the MTP head's. Do not re-quantize a weight. Do not re-represent one. Do not change the numerical format of one. Do not mirror one. This holds on disk and in memory, and it holds even when the result passes every correctness gate.

Nothing licenses a change of weight format. A lossier target substitutes a degraded model instead of optimizing the accepted one.

The MTP head is no exception. It is the organizer's pinned weights, and it is used exactly as staged. No weight on this track may be re-quantized, re-cast or mirrored, on disk or in memory. A decoder only proposes tokens, and the pinned target decides every emitted token.

What you must not change

• Everything that editablePaths does not list. Today the editable entries are ds4/ (the engine), harness/, mtp-head/ and mtp-head.manifest.json.
• fixtures/, benchmark.json, the benchmarker (benchd), the gates, the other scripts under tools/ and .github/, the tests, and the documents.
• weights/, the reference checkpoints, the scores, and the goldens.

Do not hardcode hidden prompts. Do not hardcode hidden token identifiers. Do not use timing shortcuts, protocol injection, network access, or filesystem exfiltration.

Do not add a cache keyed on a request's input tokens whose only possible hit is the harness repeating one identical computation. The benchmark measures single-pass inference. Input-independent caches stay legal. These are weights, dequantized tensors, and RoPE or mask tables keyed on shapes and offsets. Within-request KV reuse also stays legal.

Local testing vs the ranked run

The ranked run is box work. It needs the GB10 box, the staged target snapshot and the staged goldens. Off the box this repository gives you three signals.

On a machine with a GPU and the snapshot, tools/local-baseline.sh runs one leg of your engine on the public golden through the normal benchmark entry point. It measures local correctness and timing and seals no score.

On the box, tools/serve-up.sh boots and stops one resident engine, and tools/calibrate-box.sh records the box's control band. Both are organizer tools. Read each one before you run it.

WARNING — a local signal is directional, not predictive. No local signal measures the ranked composite. The ranked box run is the authority on any score.

Scoring and gates

The formula

composite = prefill_gain ^ 0.25 * decode_gain ^ 0.75

Each component is a gain:

gain = baseline_aggregate / candidate_aggregate

The score is serial-anchored. A faster candidate scores above 1.

aggregate is a sum over the pairs, per role. Each gain is therefore a ratio of sums, not a mean of per-pair ratios. The scored ranked run times the ONE prompt live_golden names, on every leg.

The pair is measured, not stored

A ranked run measures PAIRS OF LEGS. It measures them on the SAME box, in the SAME job, over the ONE prompt the fixture names in live_golden. The fixture's official_pairs sets the count, and it is 2 (David ruling 2026-09-09). Every pair is the same two legs in the same order:

The reference tree is the engine commit that the fixture pins as baseline_reference_commit. Every ranked box runs the same reference commit, so one board compares one control.

The legs run strictly one after the other, and each leg loads the model once. Per role the per-token times are summed over the pairs, and the score is the ratio of those sums. Every control leg is checked against this box's baseline calibration. All the numbers come from the same machine, minutes apart.

NOTE — the baseline is measured, not stored. No file holds a baseline pair. The goldens hold none, the fixture holds none, and the benchmarker holds none. A golden that carries benchmark.baseline_prefill_seconds_per_token or benchmark.baseline_decode_seconds_per_token is refused.

This is why each pair is measured together. A stored number describes the machine that produced it, at the temperature and on the engine of that day. The control leg describes YOUR run, on the same box, minutes before your candidate leg.

The legs never run at the same time: each leg boots its engine, is measured, and is stopped, and only then does the next leg boot. The box holds about 103.7 GiB of weights, so one engine at a time is the only arrangement that fits.

The benchmarker boots each leg's engine from that leg's own tree:

tools/serve-up.sh --boot --spec serial|mtp --draft-len N --socket-out FILE
tools/serve-up.sh --stop --socket PATH

The control leg always gets --spec serial --draft-len 0, and benchd verifies its tokens against the serial tape (--control-golden), never against the per-depth tape the candidate leg is scored on.

The per-box calibration

Each ranked box carries a calibration file. The file records what the control leg has measured on THAT box before: the mean of each axis, the spread, and a band around the mean.

The file is a health band. The benchmarker reads it to answer one question: did the control leg land where this box lands? If the control leg falls outside the band, the run stops and seals no score. A box that has changed does not produce a number.

WARNING — the calibration is never a denominator. The benchmarker divides by the control leg it just measured. It never divides by the calibration file.

The organizer writes the file with tools/calibrate-box.sh, on the box it describes. The tool runs the control leg four times on the reference tree and refuses when the spread is above 1 %. Each pass gets its own resident engine: the benchmarker boots one, measures the pass, and stops it before the next pass boots, so every pass is shaped like the single control leg a scored run measures.

The measured window

The correctness-pool row is not the scored timing. The box stages all 8 pinned prompts and the preflight verifies all 8. Each timed leg runs the one prompt live_golden names.

The parameters

The width is fixed at 1. A width other than the declared one has no certified series tag, and the benchmarker refuses it rather than run it.

BOTH FLOORS ARE 0.95 (David ruling 2026-09-09). A candidate that regresses prefill or decode by more than 5 percent is refused. The floors and the ceiling apply to the aggregate over the 2 pairs, not to one pair. The fixture declares them as decode_speedup_floor and prefill_speedup_floor, and the benchmarker enforces the fixture's values.

There is no median. Each role's per-token times are summed over the pairs, and each gain is the ratio of those sums.

kvBackend is pinned contiguous on both legs. The benchmarker refuses when it cannot honour the pinned backend. It does not degrade to another backend.

Token fidelity

The benchmarker applies a per-stream token-tolerance gate with a 10% budget.

WARNING — the gate accepts similar output, not identical output. This track does not require your output to match the serial trajectory token for token. The block-shaped forward pass diverges from the serial forward pass at near-tie argmaxes. The gate prices that divergence against the 10% budget. Do not read the gate as lossless.

The checked-in public goldens were captured on ranked hardware. A near-tie argmax can diverge on other hardware, even for correct code. Before you treat a local failure as your own regression, check whether an unmodified main fails at the same token position on your machine.

Current status

NOTE — the track is ARMED. Three statements describe the arm state.

1. fixtures/qwen3_8_125b_a6b_track.json sets official_scoring_enabled to true. That flag is the single authority on the arm state, and the benchmarker enforces it: it refuses to seal an official scoring artifact while the flag is false or absent.
2. Every one of the 8 timed-pool slots, and the hidden correctness golden, pins a real golden by {r2_path, sha256, bytes} (the correctness golden pins the live golden). tools/ranked-box-preflight.sh verifies every staged golden against its pin -- byte count then sha256 -- and refuses on any mismatch. live_golden names the one live scored prompt (botany); the other 7 rotate.
3. The launch reference candidate is stock SERIAL (num_speculative_tokens 0). It is scored against the serial control leg, which runs the same engine on the same box, so the expected launch composite is approximately 1.000 within bands (prefill +/-5 %, decode +2 % up) -- a null control of the scored pipeline. The native MTP head is the participants' improvement path (a later, separate submission).

The arm state, the pins and the serial launch reference are repository state now. Staging the goldens onto the ranked box and registering the runner remain box/organizer work. Nothing in a submission can change the arm state or the pins -- the contract fixture is trusted-side and is not an editable path.

NOTE — the track goldens are not in this repository. The 8 timed-pool tapes and the 6 per-depth oracles are organizer material. They are published in R2 at the r2_path keys the contract pins. The ranked box stages them out of band into the directory its runner service exports as MLXFAST_QWEN38_GOLDEN_DIR. tools/ranked-box-preflight.sh verifies every file there against the contract's {sha256, bytes} and refuses an extra *.json. They are never in git, so your clone does not carry them. Keep every change prompt-independent and model-general.

The organizer stages them with the signer this repository vendors:

R2_BUCKET_ENDPOINT=... R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... \
  tools/fetch-goldens.sh --all --out "$MLXFAST_QWEN38_GOLDEN_DIR"
tools/ranked-box-preflight.sh

There is ONE speculative arm on this track. allowed_modes declares serial and mtp only. DFlash was a Gemma-era second arm and is removed.

Submitting

Use the Yukon CLI for every account operation and every submission operation.

export PATH="${HOME}/.local/bin:${PATH}"

This command puts yukon on your path.

yukon login <api-key> --api <url>

This command authenticates you.

yukon clone <benchmark-id-or-name>

This command clones the benchmark repository.

Before you submit, remove build artifacts from the editable paths. The adapter under harness/ is a Cargo crate; its target/ build tree is large and is not part of a submission. The archive packages the editable-path directories as they sit on disk, so a stale target/ inflates the archive past the 25 MiB limit and the submission is rejected before it uploads.

cargo clean --manifest-path harness/protocol-adapter/Cargo.toml

This command removes the adapter's build tree. ./setup.sh and the ranked box rebuild it from source, so removing it costs you nothing.

yukon submit --model "<exact model name>" --harness "<harness name>" --note-file submission-note.md

This command uploads your editable-path archive.

yukon submissions

This command lists your submissions.

A submission archive replaces the editable paths. It rejects generated artifacts, symlinks, local scores, reference checkpoints, and any source change outside the editable surface. yukon submit does not run a local test first. No local run blocks the upload. Run the local test yourself before you submit.

The pinned artifacts

The staged snapshot holds the four target shards (111,334,654,784 bytes) and, flat beside them, the Q8_0 MTP draft head (2,786,568,256 bytes). The fixture pins all five files.

The model repository is public. It downloads without a token.

The target model

Building the engine

The scored engine is the cuda-engine adapter. ./setup.sh builds and stages it for you; to rebuild it directly:

tools/ds4/build.sh

This command builds the vendored ds4 engine, links the shim library, and builds the adapter with its real ds4 backend (--features ds4-engine). Off the box, build the adapter alone without the feature for the deterministic mock backend, which needs no GPU and no engine:

cargo build --release --manifest-path harness/protocol-adapter/Cargo.toml --bin cuda-engine

tools/stage-cuda-engine.sh

This command copies the finished cuda-engine binary to the fixed workspace path benchd resolves and spawns (.build/release/mlxfast-runtime-worker). That name is retained from the seed on purpose: it is a contract with the benchmarker, not a preference, so this repository honours the path rather than renaming it. The adapter carries no Metal kernels, so there is no mlx.metallib.

cargo is the only toolchain this repository builds with.

Continuous integration

.github/workflows/ci.yml runs on every pull request and on every push to main. It runs repository hygiene checks and the Rust adapter's cargo build/clippy/test, all on ubuntu-latest. There is no Swift build: the Swift seed package was removed in the final de-Swift.

CI is advisory. No status check is required. A red run blocks neither a merge nor a dispatch. docs/ci-coverage.md holds the detail.

CI never measures and never scores. CI holds no secret, downloads no weights, and runs no GPU test. The GPU tests and the checkpoint tests are box-only. docs/ci-coverage.md names them.

.github/workflows/benchmark.yml is the ranked pipeline. It triggers on workflow_dispatch only. It holds no secret. Its ranked job runs on [self-hosted, Linux, ARM64, qwen3.8-125b-a6b-cuda-v1] (the GB10 box). Before it measures, it verifies every box-staged asset against the contract's {sha256, bytes} pins. It publishes no score until the runner is registered, the box is staged, and the benchmarker emits a composite. Each of those gaps gives a non-zero exit and no artifact.

Where to get help

NOTE — the order of authority. The ranked box run is the authority on any score. The contract fixture fixtures/qwen3_8_125b_a6b_track.json wins over this document. This document only explains; it never overrides. If either disagrees with the benchmarker about measurement, the benchmarker wins.

License and attribution

This repository's harness code is licensed per LICENSE. The pinned checkpoint is a GGUF conversion of Qwen/Qwen3.8-Flash-Next. The model's own license terms apply to it. This repository distributes no model weights. THIRD_PARTY_NOTICES.md holds the full third-party attribution.