mooney-spark

Run Qwen3.8-Flash-Next-Mooney — a 180B-parameter MoE checkpoint compressed to 92 GB on disk / about 39 GiB in memory — on one NVIDIA DGX Spark (GB10, 121 GiB unified memory).

Two runtimes are supported; both understand the model's PQ2_0 ternary expert tensors and lowbitflash.rot.* rotation metadata:

Model weights live in a separate repository — https://huggingface.co/DJLougen/Qwen3.8-Flash-Next-Mooney (Qwen Community License 1.0). This repo is code/scripts only (MIT).

Quickstart

# on the Spark:
git clone https://github.com/DJLougen/mooney-spark && cd mooney-spark
./setup_spark.sh                      # ds4 fast runtime (default)
# or:  ./setup_spark.sh --runtime llama.cpp

# then:
~/mooney-spark/launch/serve_ds4.sh        # http://127.0.0.1:8000/v1 (OpenAI-compatible)
# or: ~/mooney-spark/launch/serve_llamacpp.sh   # http://127.0.0.1:8089/v1

HF_TOKEN is only needed while the model repo is private (or if you hit anonymous HF rate limits): export HF_TOKEN=hf_... before running setup.

setup_spark.sh is one command that:

1. Preflights the box: GB10 / compute-capability sm_121, CUDA 13.x toolkit, driver ≥ 580.159.03, ≥110 GiB free disk, MemAvailable floor, python3 + huggingface_hub.
2. Builds the ds4 engine with CUDA_ARCH=sm_121 (mandatory — upstream's default arch produces PTX that ptxas rejects on GB10). With --runtime llama.cpp it instead configures our fork with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real and builds llama-server.
3. Downloads manifest.json from the official model repo, then the 4 GGUF shards + the BF16 mmproj via huggingface_hub (resumable). Every file's size and sha256 are checked against the manifest at run time — no hashes are hardcoded, and a single mismatch aborts the setup.
4. Fetches the MTP draft head (mtp-Qwen3.8-Flash-Next.gguf, Q8_0, 2.6 GB) — it is a manifest-listed file of the Mooney repo, so it goes through the same verified download path, then is cross-checked against the cuda.fast fixture pin (size + sha256 5ff54097…). If the manifest does not list it, --mtp-source upstream falls back to the pinned unsloth source (unsloth/Qwen3.8-Flash-Next-GGUF @ 38bb39ee, path MTP/). ds4 only — the llama.cpp path doesn't use it.
5. Writes a launch script that starts the server bound to 127.0.0.1 inside scripts/memguard.sh, a MemAvailable floor guard tuned for the Spark's unified memory (ds4 launcher: start floor 75 GiB, soft 20, hard-kill 10 — sized so a full 256k request fits, whose worst measured dip was ≈49 GiB; llama.cpp launcher: 60/30/22 — override with MG_* env vars). The ds4 launcher defaults to -c 262144 (full context; DS4_CTX overrides).

Other modes:

./setup_spark.sh --dry-run                 # print every step, change nothing
./setup_spark.sh --verify-only DIR --manifest manifest.json   # hash-check only
./setup_spark.sh --verify-only DIR --expect file.gguf=SHA256  # pin list, no manifest

The ds4 engine ships as a standalone public fork — DJLougen/cudafast-qwen38-125b-a6b-engine (branch lbf/pq2-rot) — which setup_spark.sh clones by default; DS4_SRC_DIR accepts any existing checkout, and a vendored subtree at engine/cudafast/ is auto-detected. The llama.cpp fallback comes from DJLougen/prism-llama.cpp (branch lbf/flashnext-ternary, LLAMA_GIT_URL/LLAMA_SRC_DIR).

Engine checkouts that setup_spark.sh clones itself (under engine/) are always checked out to the release-pinned SHAs — on first clone and on every re-run, so a later push to the branch never silently changes what you build. Trees with tracked local edits are refused rather than clobbered (commit or stash them, or set ALLOW_UNPINNED=1 to keep a managed tree wherever it is). DS4_SRC_DIR/LLAMA_SRC_DIR checkouts are never moved; the script warns with the exact pin command when they are not at the pin.

Where the weights go

Everything installs under $HOME/mooney-spark/ (override with INSTALL_DIR, MODEL_DIR, ENGINE_DIR):

~/mooney-spark/
  models/   Qwen3.8-Flash-Next-Mooney-PQ2_0-0000{1..4}-of-00004.gguf
            mmproj-Qwen3.8-Flash-Next-Mooney.gguf            (vision, BF16)
            mtp-Qwen3.8-Flash-Next.gguf                      (MTP head, Q8_0; ds4)
            .manifest/manifest.json
  engine/   cudafast/  or  prism-llama.cpp/
  launch/   serve_ds4.sh  /  serve_llamacpp.sh

Expected memory and speed (measured, DGX Spark)

ds4: engine c9e0679f through ds4-server with this launcher's settings, greedy (temperature 0), median of 3 runs that each generate exactly 128 tokens; held-out documents with a question at the end; 256k = a 256,457-token prompt. Greedy MTP output is token-identical to MTP-off on 15/15 test prompts up to 128k context. --mtp-draft 2 (draft depth 1) measures 48.9 / 43.0 / 39.0 / 38.9 / 37.7 / 36.9 tok/s at the same contexts; set DS4_MTP_DRAFT=2 to use it. MTP only speeds up greedy requests: sampled requests (temperature > 0, including the server default) run at the no-MTP speed. Memory = MemAvailable drop at load (the 32k value is from the earlier campaign). llama.cpp numbers are the model card's measured values. For comparison, Unsloth's 4-bit UD-Q4_K_XL build on the stock cuda.fast engine measured 30.4 / 27.3 tok/s (short / 4k) without MTP on the same harness, so Mooney's advantage is memory, not single-stream speed.

Licensing

• This repo's scripts/docs: MIT (LICENSE).
• ds4 engine: MIT — Layr-Labs/cudafast-qwen38-125b-a6b-engine vendoring Layr-Labs/ds4 (which descends from antirez/ds4); our delta is branch lbf/pq2-rot of DJLougen/cudafast-qwen38-125b-a6b-engine. See engine/README.md.
• llama.cpp fork: MIT (ggml-org/llama.cpp + PrismML's fork); our delta is branch lbf/flashnext-ternary of DJLougen/prism-llama.cpp.
• Model weights are NOT in this repo — they are downloaded from the official HF repo at setup time and are under the Qwen Community License 1.0.
• The MTP head ships inside the Mooney model repo (mtp-Qwen3.8-Flash-Next.gguf, Q8_0, a manifest-listed file, Qwen Community License 1.0). The cuda.fast-pinned unsloth source remains a documented fallback (--mtp-source upstream).

Credits

• Layr-Labs — the cuda.fast ds4 engine this repo's fast runtime ports.
• ds4 authors / antirez — the original ds4 engine (MIT).
• PrismML — the llama.cpp fork and the Bonsai quantization line Mooney's recipe builds on.
• ggml-org — llama.cpp (MIT).
• Zach Mueller and Lambda — the compute that made this release possible.
• Qwen team — Qwen3.8-Flash-Next base checkpoint.